HMD transitions for focusing on specific content in virtual-reality environments

ABSTRACT

Methods and systems for presenting an object on to a screen of a head mounted display (HMD) include receiving an image of a real-world environment in proximity of a user wearing the HMD. The image is received from one or more forward facing cameras of the HMD and processed for rendering on a screen of the HMD by a processor within the HMD. A gaze direction of the user wearing the HMD, is detected using one or more gaze detecting cameras of the HMD that are directed toward one or each eye of the user. Images captured by the forward facing cameras are analyzed to identify an object captured in the real-world environment that is in line with the gaze direction of the user, wherein the image of the object is rendered at a first virtual distance that causes the object to appear out-of-focus when presented to the user. A signal is generated to adjust a zoom factor for lens of the one or more forward facing cameras so as to cause the object to be brought into focus. The adjustment of the zoom factor causes the image of the object to be presented on the screen of the HMD at a second virtual distance that allows the object to be discernible by the user.

CLAIM OF PRIORITY

This application claims priority under 35 U.S.C. § 119 to U.S.Provisional Patent Application No. 62/349,652, filed on Jun. 13, 2016,and entitled “Methods and System for Rendering Objects in AugmentedReality”, the disclosure of which is incorporated herein by reference inits entirety.

FIELD

The present disclosure relates to systems and methods for presentingobjects using augmented reality, on a head mounted display.

BACKGROUND Description of the Related Art

Computing and video gaming industry have seen many changes over theyears. As computing power has expanded, developers of variousinteractive applications, such as video game applications, have createdapplication software that takes advantage of these increases incomputing power. To this end, application developers, such as video gamedevelopers, have been developing games that incorporate sophisticatedoperations to increase interaction between a user and the gaming systemso as to produce a very realistic game play experience.

Generally speaking, gesture input refers to having an electronic device,such as a computing system, video game console, smart appliance, etc.,react to some gesture made by the player and captured by the electronicdevice. One way of accomplishing a richer interactive experience is touse wireless game controllers to provide gesture input. Movement of thewireless game controllers are tracked by the gaming system in order totrack a player's movements and to determine gesture input provided bythe player, and use these movements and gesture inputs as inputs foraffecting the state of the game.

Another way of accomplishing a more immersive interactive experience isto use a head-mounted display. A head-mounted display (HMD) is worn bythe player and can be configured to present various graphics, such as aview of a virtual scene, on a display screen of the HMD. The graphicspresented on the screen of the head-mounted display can cover a largeportion or even all of a player's field of view. The head-mounteddisplay can provide a visually immersive experience to the player byblocking the view of the real-world scene.

To further enhance the immersive experience for a player at any giventime, the HMD may be configured to just render game scene of a virtualgame generated by a computer/computing device, for example, or liveimages from a real-world environment, or a combination of both thereal-world environment and virtual game scene. However, all the objectsor details provided in the images of the real-world environment that arerendered at the display screen of the HMD may not be completelydiscernible by the player.

It is within this context that embodiments of the invention arise.

SUMMARY OF THE INVENTION

Embodiments of the present invention disclose methods, systems andcomputer readable media that are used for presenting an object from areal-world environment by bringing the object into focus on a screen ofa head mounted display (HMD) for a user to view. In someimplementations, the HMD is configured to receive images from areal-world environment or a virtual reality (VR) scene and render theimages on a screen of the HMD. While rendering the images, informationprovided by one or more sensors in the HMD is used to detect a gazedirection of a user wearing the HMD. In response to detecting the gazedirection, an object from the images of the real-world environment thatis in line with the gaze direction of the user is identified, and asignal is generated to adjust a zoom factor for the lens of the one ormore forward facing cameras disposed on the HMD. The zoom factoradjustment causes an image of the real-world environment that includesthe object to be brought into focus when rendered on the screen of theHMD. In some implementations, the HMD may take into consideration visioncharacteristics of the user's eyes when determining the zoom factor forthe lens of the HMD. In some implementations, the sensors determine ifthe user has been gazing at the object for at least a pre-definedthreshold period before generating the signal for adjusting the zoomfactor for the lens of the HMD. In some implementations, the signal thatincludes instructions to adjust zoom factor of the image may includeinformation to adjust optical setting of the lens so as to allow opticalzooming of the image of the object. In other implementations, the signalthat includes instructions to adjust zoom factor may include instructionto perform digital zooming of the image. Thus, depending on the type ofimages that are being rendered, the zoom factor may include instructionsor information to allow either optical zooming or digital zooming.

The implementations provide ways to present augmented reality of areal-world environment by allowing specific points or specific ones ofobjects captured in the images of the real-world environment to bemanipulated and presented to the user so that the user can discern thespecific points or objects clearly. The embodiments may be extended topresent an enhanced view of a VR scene rendered on a screen of the HMDby bringing a specific object of the VR scene into focus. In someembodiments, a signal that is generated may take the visioncharacteristics of the user's eyes into consideration when adjusting thezoom factor. Augmenting the real-world environment or enhancing the VRscene allows the user wearing the HMD to be able to discern the variousobjects or portions of a scene that is being rendered on the screen ofthe HMD to have a satisfying immersive experience.

Images from the VR scene are usually pre-recorded images/videos. In someembodiments, when images from a VR scene are being rendered at the HMD,historic input from other users that have viewed the images of the VRscene may be considered when generating the signal to adjust the zoomfactor. For example, in some embodiments, the historic inputs from otherusers may be correlated with content from the VR scene to determinespecific zoom factor settings that caused other users to experiencedizziness, motion sickness, etc. This information may be used to refinethe zoom factor adjustment for the lens of the one or more forwardfacing cameras when rendering such content, so that the user does notexperience motion sickness, dizziness, etc., when images from the VRscene are presented to the user wearing the HMD. In some embodiments,options may be provided to the users (e.g., thrill seekers) to overridesuch refinement in order to be able to view the content with specificzoom factor settings.

The embodiments of the invention enable the HMD to act as virtualbinoculars by allowing the HMD to present specific portions of images ofthe VR scene or the real-world environment to be zoomed in by adjustingthe zoom factor of the lens of the image capturing devices. Inputs fromthe various controllers and sensors of the HMD can be used todynamically activate or deactivate specific settings of the lens of theone or more forward facing cameras to allow the user to view the contentwith sufficient clarity. In alternate implementations, specific featuresor portions of an image may be adjusted to allow an object that includesthe specific features or in the portions is to be brought into focus. Insome implementations, the HMD acts to virtually teleport a user from onelocation within the VR scene or the real-world environment to another,and such teleporting is done to enhance the user's experience of the VRscene or augmented reality (AR) world. For example, a specific area or aportion of the VR scene may be brought into focus to make it appear thatthe user is teleported to a location proximal to the specific area orportion.

In one embodiment, a method for presenting an object from a real-worldenvironment on to a screen of a head mounted display (HMD), is provided.The method includes receiving an image of a real-world environment inproximity of a user wearing the HMD. The image is received from one ormore forward facing cameras of the HMD and processed for rendering on ascreen of the HMD by a processor within the HMD. A gaze direction of theuser wearing the HMD, is detected using one or more gaze detectingcameras of the HMD that are directed toward one or each eye of the user.Images captured by the forward facing cameras are analyzed to identifyan object captured in the real-world environment that correlates withthe gaze direction of the user. The image of the object is rendered at afirst virtual distance that causes the object to appear out-of-focuswhen presented to the user. A signal is generated to adjust a zoomfactor for lens of the one or more forward facing cameras so as to causethe object to be brought into focus. The adjustment of the zoom factorcauses the image of the object to be presented on the screen of the HMDat a second virtual distance that allows the object to be discernible bythe user.

In some implementations, the signal is generated after determining theuser's gaze direction is directed toward the object for a pre-definedlength of time.

In some implementations, the object is brought into focus for apre-defined period of time. Upon expiration of the pre-defined period oftime, images from the real-world environment are rendered.

In some implementations, the zoom factor is adjusted to account forvision characteristics of the user's eyes wearing the HMD.

In some implementations, the signal to adjust the zoom factor includes asignal to adjust an aperture setting of lens in the one or more forwardfacing cameras so as to cause an adjustment to a depth at which theimage of the object is captured by the one or more forward facingcameras of the HMD.

In some implementations, the signal to adjust the zoom factor includes asignal to adjust a focal length of lens of the one or more forwardfacing cameras when capturing the images of the object in the real-worldenvironment. The adjustment to the focal length of the lens causes azooming in on the object.

In some implementations, the signal to adjust the zoom factor includes asignal to control a speed of zooming, wherein the speed of zooming iscontrolled based on type of the images of the real-world environmentcaptured or based on the user.

In some implementations, the signal to adjust the zoom factor furtherincludes a signal to adjust brightness level of the screen of the HMD.

In some implementations, a three-dimensional digital model of thereal-world environment is constructed using the image captured by theforward facing cameras of the HMD. The three-dimensional digital modelis constructed by tracking different points captured in multiple framesof the images using more than one camera and correlating the differentpoints captured by the cameras to a three-dimensional space.

In some implementations, identifying the object includes outlining theobject captured in the images that correspond with the gaze directionand receiving confirmation of the object from the user wearing the HMD.

In some implementations, a method for rendering an object on a screen ofa head mounted display (HMD), is disclosed. The method includesreceiving images from a virtual reality (VR) scene for rendering on thescreen of the HMD. Selection of an object from the images of the VRscene rendered on the screen of the HMD, is detected. The selectedobject is determined to be rendered at a first virtual distance thatmakes the object appear to be out of focus for a user wearing the HMD.An area in the VR scene that is proximal to the object, is identified.The identified area defines a freedom of movement for the user inrelation to the object when viewing the object. A signal is generated tovirtually teleport the user to the area proximal to the selected object.The virtual teleportation of the user causes the object to be presentedon the screen of the HMD at a second virtual distance that allows theobject to be discernible by the user.

Other aspects and advantages of the invention will become apparent fromthe following detailed description, taken in conjunction with theaccompanying drawings, illustrating by way of example the principles ofthe invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention, together with further advantages thereof, may best beunderstood by reference to the following description taken inconjunction with the accompanying drawings.

FIG. 1 illustrates a simplified block diagram of a communicationarchitecture of a head mounted display that is used in bringing anobject captured from a real world or virtual world into focus, inaccordance with an embodiment of the present invention.

FIG. 2 illustrates specific ones of the modules of the HMD that are usedin generating a signal to adjust a zoom factor of one or more forwardfacing cameras of the HMD, in accordance to an embodiment of the presentinvention.

FIG. 3 illustrates example architecture of a system that is used toadjust an image of an object from a real world environment to bediscernible to a user wearing the HMD, in accordance to an embodiment ofthe present invention.

FIGS. 4A-4C illustrate various stages involved in bringing an objectinto focus in response to detecting a gaze direction of a user, in someembodiments of the invention.

FIGS. 5A and 5B illustrate an image of a real-world environment that isbeing presented on a screen of the HMD and teleporting of a user todifferent locations within the real-world environment captured by theHMD, based on detected gaze direction of a user, in some embodiment ofthe invention.

FIG. 6 illustrates a graphical representation of screen transitionduring presentation of an enhanced view of a real-world environment orvirtual world scene, in some embodiments of the invention.

FIG. 7 illustrates operation flow of a method that is used for providingimage of an object in a real-world environment or from a virtual scene,in accordance to an embodiment of the invention.

FIG. 8 illustrates example Information Service Provider architecture fordelivering informational content and services to users who aregeographically dispersed and connected via network, in accordance withone embodiment of the present invention.

FIG. 9 illustrates a simplified block diagram of an example Game System,in accordance with various embodiments of the invention.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth inorder to provide a thorough understanding of the present invention. Itwill be apparent, however, to one skilled in the art that the presentinvention may be practiced without some or all of these specificdetails. In other instances, well known process steps have not beendescribed in detail in order not to obscure the present invention.

According to various implementations, a user wearing a head mounteddisplay (HMD) may be presented with images of a real-world environmentor a virtual reality (VR) scene on a screen of the HMD. The image of thereal-world environment may be captured by one or more forward facingcameras of the HMD. Alternately, image from the VR scene may be part ofa video game or may be a pre-recorded video that is transmitted from acomputing device that is communicatively connected to the HMD. Thevarious components (e.g., sensors, cameras, etc.,) in the HMD are usedto detect a gaze direction of a user wearing the HMD. The gaze directionof the user is used to identify an object within the images that is ofinterest to the user or has captured the attention of the user. Upondetecting the user's continued interest in the object, a signal isgenerated by the processor of the HMD to adjust optical elements of theHMD to perform an optical zoom or to adjust images via a digital zoom sothat the object that is of interest to the user or has captured theattention of the user is brought into focus.

In some implementations, the HMD acts as a virtual teleporter byadjusting how the images are rendered on the screen so as to make itappear that the user wearing the HMD has been dynamically teleported toan area closer to an object or area or a scene that has captured theinterest of the user. The speed of teleporting or transitioning of theuser may depend on the type of content that is being rendered and canalso be adjusted to ensure that such transition does not cause anydistress to the user. Once the user has been virtually transitioned tothe area closer to the object or scene, the user may provide input usinga controller or through gestures, and such input provided in thephysical or virtual world is interpreted and used to adjust content thatis being rendered on the screen of the HMD. In some implementations, thearea closer to the object is correlated with a physical environment inwhich the user operates his HMD. In such embodiments, a boundary of thearea closer to the object to which the user has been teleported iscorrelated with the confines of the user's physical world environment.As the user interacts with the images rendered on the HMD, suchinteractions are interpreted in relation to the real-world environmentby the HMD system and appropriate feedback is provided to the user. Forexample, if the teleporting causes the user to be at the edge of a scene(e.g., edge of a virtual cliff, a virtual building, a physical wall, areal world obstacle, etc.) or be in a situation that can potentiallyharm the user or cause the user to feel disoriented, etc., the feedbackmay provide appropriate warning. The feedback may be provided in visualformat, audio format, haptic format, textual format, optical format, orany combinations thereof.

With the general understanding of the invention, specific embodimentswill be described with reference to the various drawings. It should benoted that various embodiments described in the present disclosure maybe practiced without some or all of the specific details describedherein. In other instances, well known process operations have not beendescribed in detail in order not to unnecessarily obscure variousembodiments described in the present disclosure.

FIG. 1 is an embodiment of an exemplary configuration of a system 100used to implement the various embodiments. The system includes a headmounted display (HMD) 104 that is worn on a head of a user, and acomputer 172. In some implementations, the system may also include ahand-held controller (HHC) 106 to allow the user to interact withcontent provided for rendering on a display screen of the HMD 104 andgenerate user input. In various implementations, the computer 172 may bea general purpose computer, a special purpose computer, a gamingconsole, a mobile phone, a tablet device, or other such device, which isconfigured to execute one or more portions of an interactiveapplication, such as a video game, that provides content for renderingon the display screen of the HMD 104. The interactive application may bea multi-user game application 117 that is played by multiple users ormay be a single user game application played by the user. In someembodiments, at least a portion 117 b of the game application isexecuting on the computer. In such embodiments, any remaining portions117 a of the interactive game application may be executed on a cloudsystem, such as a game cloud system 102, e.g., on one or more virtualmachines (VMs), wherein the game content and user interactions areexchanged through a network 110, such as the Internet. In someembodiments, the computer 172 may be part of the game cloud system 102and the HMD and the hand-held controller (HHC) directly communicate withthe computer disposed on the game cloud system 102 via the network 110.In such embodiments, a portion of the interactive application isexecuted by the computer on the game cloud system and the remainingportion of the interactive application is executed on the HMD 104.

The HMD 104 is a device, worn directly on a head of a user or as part ofa helmet. The HMD 104 has a small display optic, e.g., lens, glass,waveguide, etc., in front of one or each eye of the user. In someembodiments, a scene, e.g., virtual scene, augmented virtual realityscene, augmented real world environment, etc., is displayed on a displayscreen of the HMD and is viewed through the display optics provided infront of one or each eye of the user. In instances where the displayoptics are in front of each eye of the user, both eyes see one scene.

In some embodiments, the HMD 104 is capable of receiving and renderingvideo output from an application executing on the computer and/or on thecloud system, such as a game cloud system. In various embodiments, theHHC and/or the HMD communicates wirelessly with the computer, as thisprovides for greater freedom of movement of the HHC and/or the HMD thana wired connection. In an alternate embodiment, the HHC and/or the HMDcommunicate with the computer through a wired connection.

The HHC 106 may include any of various features, such as buttons,inertial sensors, trackable LED lights, touch screen, a joystick withinput controls, directional pad, trigger, touchpad, touchscreen, and mayhave circuitry/logic to detect and interpret hand gestures, voice inputor other types of input mechanisms for providing input to theinteractive application. Furthermore, the HHC may be a motion controllerthat enables the user to interface with and provide input to theinteractive program by moving the controller.

Along similar lines, the HMD 104 may include a user input circuit thatenables the user to interface with and provide input to the interactiveapplication by moving the HMD. Various technologies may be employed todetect the position and movement of a the HMD and/or motion controllerthat is communicatively coupled to the HMD. For example, the motioncontroller and/or the user input circuit of the HMD may include varioustypes of inertial sensor circuits, such as accelerometers, gyroscopes,and magnetometers. In some embodiments, the motion controller mayinclude global position systems (GPS), compass, etc. In someembodiments, an accelerometer is a 6-axis low latency accelerometer. Insome embodiments, the motion controller and/or the user input circuitcan include one or more fixed reference objects (otherwise termed“marker elements”), e.g., light emitting diodes (LEDs), colored points,light reflectors, etc., that can be tracked using image capturingdevices. For example, the images of the fixed reference objects arecaptured by one or more digital cameras (not shown) of the system thatare disposed to face a user wearing the HMD and to track the position ofthe user, the HMD and/or the HHC. Gesture actions and movement of theuser, the HMD and/or the HHC are also captured by the digital cameras.In some embodiments, a digital camera includes a video camera thatfurther includes a single Charge Coupled Device (CCD), an LED indicator,and hardware-based real-time data compression and encoding apparatus sothat compressed video data may be transmitted in an appropriate format,such as an intra-image based motion picture expert group (MPEG) standardformat. The position and movement of the user, the motion controllerand/or the HMD can be determined through analysis of the images capturedby the one or more digital cameras.

In one embodiment, the HMD 104 includes a router 152 to communicate withthe internet 110. In an alternate embodiment, the HMD 104 maycommunicate with the cloud system over the Internet 110 using a router152 that is external to the HMD 104. In some embodiments, the game cloud102 is referred to herein as a game cloud system. The HMD 104 is placedby a user on his head in a manner that is similar to the user putting ona helmet, so that lenses of the HMD 104 are located in front of one orboth eyes of the user. In some embodiments, the HMD 104 is worn likeglasses, (e.g., prescription glasses, goggles, etc.). The HHC 106 isheld by the user 106 in his/her hands.

In various embodiments, instead of the HHC 106, hands of the user 108may be used to provide gestures, e.g., hand gestures, finger gestures,etc., that may be interpreted by interactive application and/or thelogic within the HMD 104. In some embodiments, the user may wear aninteractive glove with sensors to provide tactile feedback. Theinteractive glove acts as the HHC, when worn by a user, and providesinput in the form of interactive gestures/actions to the interactiveprogram and/or the HMD. Similar to the HHC, the interactive glove mayinclude marker elements, such as LEDs, light reflectors, etc., to allowdetection of various movements. The interactive glove is one form ofwearable device that is used to provide input to the HMD and/or theinteractive program and that other forms of wearable clothing/device mayalso be engaged. A digital camera 101 of the HMD 104 captures images ofthe gestures provided by a user and a processor within the HMD 104analyzes the gestures to determine whether a game displayed within theHMD 104 is affected by the gestures.

In one embodiment, the digital camera 101 is located on a face plate ofthe HMD 104 facing forward to capture real-world images includinggestures provided by the user. In some embodiments, more than onedigital cameras may be provided on the face plate of the HMD 104 tocapture different angles of the real-world images. In some embodiments,the digital camera may be a stereo camera, an IR camera, a single-lenscamera, etc. As used herein, the processor of the HMD may be amicroprocessor, a programmable logic device, an application specificintegrated circuit (ASIC), or a combination thereof.

The system 100 includes a network 110, which may be a local area network(LAN), a wide area network (WAN), or a combination thereof. Examples ofthe network 110 include the Internet, an Intranet, or a combinationthereof. In some embodiments, the network 110 uses a transmissioncontrol protocol (TCP)/Internet Protocol (IP) or a user datagramprotocol/IP (UDP/IP) to communicate media data via the network 110between the game cloud 102 and the HMD 104 or the HHC 106. Theembodiments are not restricted to the TCP/IP or UDP/IP protocol but canalso engage other forms of communication protocols (including anyproprietary or non-proprietary protocols) for communicating media datavia the network. In various embodiments, the network uses a combinationof Ethernet and TCP/IP protocol to communicate media data via thenetwork 110 between the game cloud 102 and the HMD 104 or the HHC 106.

The game cloud 102 includes a coder/decoder (codec) 112 and a streambuffer 114. The stream buffer 114 stores a stream of media data 116,which is generated upon execution of a game program 117. The media data116 includes virtual environment data, virtual game object data, acombination thereof, etc. The virtual environment data is used togenerate a virtual environment of a game and the virtual game objectdata is used to generate one or more virtual game objects, e.g., virtualgame characters, virtual game objects, virtual points, virtual prizes,game interface, etc. Examples of a virtual environment include a virtualgeographic region, e.g., a virtual city, a virtual road, a virtual lake,a virtual ocean, etc. The game program 117 is an example of theinteractive application executed by one or more servers of the gamecloud 102. The codec 112 uses a compressor/decompressor to code/decodemedia data using lossy compression, lossless compression, etc.

The HMD 104 is used to access an operating system (OS) that is executedby the processor of the HMD 104. For example, selection and activationof a button in the HMD 104 enables the processor of the HMD 104 toexecute the OS. Similarly, the HHC 106 may be used to access an OS thatis executed by the processor of the HHC 106. A button on the HHC 106 maybe used to have the processor of the HHC 106 to execute the OS.

In some embodiments, the OS allows the HMD 104 to directly access thenetwork 110. For example, a user may select a network access applicationthat is executed by the processor of the HMD 104 on top of the OS, usinga network access icon, a network access symbol, etc. The network accessapplication provides a list of networks from which to select a networkfor accessing the network 110. User authentication may be required toaccess the network 110, in accordance to network access protocol. Accessto the network 110 is enabled for the user upon selection and successfuluser authentication (if needed). A built-in router (not shown) withinthe HMD 104 uses the network 110 to interact with the game cloud toexchange game data. In these embodiments, the communication between thenetwork 110 and the HMD 104 follows a wireless communication protocol.Along similar lines, the HHC 106 gains access to the network 110 byselecting the network using network access application and thecommunication between the HHC 106 and the network follows a wirelesscommunication protocol.

Once the network 110 is accessed, the OS allows the HMD 104 to accessthe game program 117 in a manner similar to the selection of thenetwork. For example, when the user selects a game access applicationexecuted by the processor of the HMD 104 on top of the OS through a gameaccess icon, a game access symbol, etc., the game access applicationrequests access to the game program 117 via the network and theprocessor of the HMD 104 for displaying to the user.

Upon obtaining access to the game program 117, a microcontroller of theHMD 104 displays game scenes of the game on a display screen of the HMD104. In some embodiments, the display screen of the HMD 104 is a highperformance miniature screen to reduce blur when the HMD 104 is movedrapidly. In one embodiment, the display screen is a Liquid CrystalDisplay (LCD) screen, liquid crystal on silicon (LCoS), or organic lightemitting diodes (OLEDs), or cathode ray tubes, etc. Images are projectedby the lens of the display optics of the HMD onto the display screens.Adjustment may be made to the lens of the display optics or the displayscreens and such adjustments affect the images that are rendered on thedisplay screens of the HMD. The user performs one or more head and/oreye motions, e.g., head tilting, winking, gazing, shifting gaze,staring, etc., and each head or eye motion triggers the user inputcircuit to generate an input, which may be used to play the game. Inthese embodiments, the game program 117 executes on the game cloud 102and the communication between the game program 117 and the HMD 104 isthrough the built-in router and the network 110. The images rendered onthe display screen of the HMD 104 are viewed through the display optics,which provides near-eye focus.

In some embodiments, the game access application requests userauthentication information, such as a username and/or a password, fromthe user to access the game program 117. Upon receiving successfulauthentication from the game cloud 102, the game access applicationallows access of the game program 117 to the user.

In various embodiments, instead of accessing the gameapplication/program, the user requests access to a web page uponaccessing the network 110 and the web page allows the user to access thegame program 117. For example, the user selects a web browserapplication via the user input circuit or via the HHC 106 to access aweb page. Upon accessing the web page, the user plays a game displayedon the web page or accesses the game using a link provided within. Thegame is rendered when the game program 117 is executed on the game cloud102. In some embodiments, user authentication may be required beforeproviding access to the web page to play the game that is displayed whenthe game program 117 is executed on the game cloud 102. The usernameand/or the password is authenticated in a manner similar to thatdescribed above when the user accesses a game via the game accessapplication.

When the game program 117 is accessed, the codec 112 encodes, e.g.,compresses, etc., a digital data stream of the media data 116 forsending a stream of encoded media data to the HMD 104 via the network110. In some embodiments, a digital data stream of the encoded mediadata is in the form of packets for sending via the network 110.

The HMD 104 receives the digital data stream of the encoded media datafor the selected game program via the network 110 from the codec 112 andthe digital data stream is processed, e.g., de-packetized, decoded,etc., and the processed stream is used to display game scene on thedisplay screen of the HMD 104. As a game scene is being displayed on thedisplay screen of the HMD 104, an external camera 101 of the HMD 104captures one or more images of a real-world environment in the immediatevicinity of the user, from a user's perspective. In some embodiments,the external camera 101 is a video camera. Examples of the real-worldenvironment include a room from where the user is accessing the game, ageographical region in which the user is located, real-world objectsaround the user, etc. Examples of a geographical region include a park,a road, a street, a lake, a city, a landmark, etc. Examples of areal-world object include a bus stand, a coffee shop, a store, anoffice, a vehicle, a room, a desk, a table, a chair, a ball, etc.Real-world environment data, including one or more images of thereal-world environment, in one embodiment, is processed and storedlocally in the HMD and used for subsequent rendering on the screen ofthe HMD. User input may be processed, packetized and encoded by the HMD104, and sent to the codec 112 in the game cloud 102 through thebuilt-in router and the network 110. In some embodiments, in addition tothe user input, the real-world environment data may also be packetizedand encoded by the HMD 104 and sent as a stream of encoded environmentdata via the built-in router of the HMD 104, the network 110 to thecodec 112 in the game cloud 102.

User inputs may be provided through the HMD 104 and/or the HHC 106. Forexample, the user may provide input using input interface/mechanismprovided in the HMD 104. Alternately, the user may perform hand motions,e.g., press of a button, movement of a joystick, hand gesture, fingergesture, a combination thereof, etc., using the HHC and such user inputprovided at the HHC 106 generates input data that is converted intoinput signals by a communications circuit of the HHC 106 for sending toa communications circuit of the HMD 104. Of course, the HHC includeshand-held controllers, joysticks, motion controllers, wearable articlesof clothing, wearable devices, etc. The input signals originating fromthe HHC 106 and the HMD 104 are converted from an analog form to adigital form by the communications circuit of the HMD 104, packetized,encoded by the HMD 104 and sent via the network 110 to the codec 112.Examples of a communications circuit of the HMD include a transceiver, atransmit/receive circuitry, a network interface controller, etc.

In some embodiments, the game program 117 maps input data that isgenerated by the HMD with input data that is generated at the HHC (fore.g., based on the hand motions) to determine whether to change a stateof a game that is displayed on the HMD 104. For example, when an inputfrom the HMD is received via the network 110 with an input generated atthe HHC 106, such as a press of a button on the HHC 106, the gameprogram 116 determines to change a state of a game. Otherwise, the gameprogram 117 determines not to change a state of a game.

Input data of the inputs generated based on the hand motions and/orhand-held controller motions are communicated by a communicationscircuit of the HHC 106, e.g., a transceiver, a transmit/receivecircuitry, etc., to a communications circuit, e.g., a transceiver, atransmit/receive circuitry, etc., of the HMD 104. Input datacommunicated to the HMD and/or input data generated by the HMD arepacketized and encoded by the HMD 104 and sent as a stream of encodedinput data via the network 110 to the codec 112. For example, the inputdata may be sent directly from the HMD 104 using the built-in router viathe network 110 to the game cloud 102. In a number of embodiments, theuser performs the hand motions and provides user input from the HMD tochange a location and/or orientation of the virtual object rendered atthe HMD.

The codec 112 decodes, e.g., decompresses, etc., the stream of encodedinput data received via the network 110 from the HMD 104 and the decodedinput data is buffered in the stream buffer 114 for de-packetizing andsending to the game program 117. One or more servers of the game cloud102 de-packetizes the stream of decoded input data and sends the inputdata to the game program 117. Upon receiving the input data, the gameprogram 117 generates next media data that is packetized by one or moreservers of the game cloud 102 to generate a stream of next media data.The additional media data may include modifications to game play,including modifications to virtual game object, e.g., computer-generatedobject, etc., that is used for updating the virtual game environmentrendered on the HMD. The stream of next media data is stored in thestream buffer 114, encoded by the codec 112, and sent as a stream ofencoded next media data via the network 110 to the HMD 104. The HMD 104receives the stream of encoded next media data, de-packetizes thestream, and decodes the encoded next media data to provide the nextmedia data to the microcontroller of the HMD 104. The microcontroller ofthe HMD 104 changes a display of a game scene rendered on the screen ofthe HMD based on the next media data. For example, the microcontrollerchanges a look, position, and/or orientation of the virtual game objectthat is either overlaid on the one or more images of the real-worldenvironment or simply rendered on the screen of the HMD 104. It shouldbe noted that the input data generated at the HHC and/or the HMD changesa state of the game. In some embodiments, a display of a game scene isreferred to herein as a portion of interactivity associated with thegame program 117.

In various embodiments, instead of communicating the input data that isgenerated based on the hand motions from the HHC 106 to the HMD 104, theinput data is communicated directly from the HHC 106 via the network 110to the codec 112. The input data that is generated at the HHC 106 iscommunicated by the HHC 106 in a manner similar to the communication bythe HMD 104. For example, the input data that is generated based on thehand motions from the HHC 106 is encoded and packetized by the HHC 106and sent as a stream of encoded input data via the network 110 to thecodec 112.

It should be noted that in the embodiment illustrated in FIG. 1, the HMDand the HHC individually directly communicate with the network 110without going through a game console or an external router. In analternate embodiment, the HHC may communicate with the HMD to transmitthe input data generated at the HHC and the HMD may directly communicatethe data originating from the HHC and/or the HMD with the network 110.In both of these embodiments, media data 116, additional media data, thenext data, etc., are streamlined directly to a wireless access card(WAC) of the HMD 104 by the codec 112 of the game cloud 102 via thenetwork 101 and the built-in router. Moreover, in these embodiments,data, e.g., input data, real-world environment data, etc., is streameddirectly by the WAC of the HMD 104 to the codec 112 of the game cloud102 via the built-in router and the network 110. The WAC in conjunctionwith the built-in router of the HMD is able to transmit the streamingmedia data and the input data to and from the HMD.

In some embodiments, a separate router 152 is provided between the HMD104 and the network 110. The router 152 also acts as an interfacebetween the HHC 106 and the network 110. In this embodiment, the WAC ofthe HMD 104 will interface with the router 152 to communicate with thenetwork 110. In some embodiments, the HMD 104 is coupled to the router152 via a wireless connection, e.g., a Bluetooth connection or a Wi-Ficonnection, etc. Moreover, the HHC 106 is coupled to the router 152 viaa wireless connection. In some embodiments, the router 152 is coupled tothe network 110 via a wired connection. When a router is provided, astream of encoded data is sent from the HMD 104 or the HHC 106 to therouter 152. The router 152 routes, e.g., directs, etc., the stream ofencoded data to a path in the network 110 to facilitate sending thestream to the codec 112 on the game cloud. The router 152 uses the IPaddress of the codec 112 to route the stream of encoded data to thecodec 112. In some embodiments, the router 152 determines a network pathof the network 110 based on network traffic factor, e.g., packet trafficon the network path, congestion on the network path, etc.

The router 152 receives a stream of encoded data from the game cloud 102via the network 110 and routes the stream of encoded data to the HMD104. For example, the router 152 routes the stream of encoded datareceived from the game cloud 102 via the network 110 to the HMD 104based on the IP address of the HMD 104. In some embodiments that use thesystems 100, the game execution occurs mostly on the game cloud 102. Insome embodiments, some part of the game may execute on the HMD 104 whilethe remaining portions may execute on the game cloud 102.

In some embodiments, a list of wireless networks is rendered on thescreen of the HMD 104 for user selection. Alternately, in some otherembodiments, a list of wireless networks is presented on a displayscreen associated with the computer 172. For example, when the computer172 is a mobile phone, the mobile phone includes a display screen fordisplaying the list of wireless networks. As another example, when thecomputer 172 is coupled to a television display screen, the list ofwireless networks is displayed on the television display screen. Inthese embodiments, the list of wireless networks is accessed when theprocessor of the computer 172 executes the wireless access applicationstored within a memory device of the computer 172 to access the network110. The processor 176 executes the wireless access application when theuser generates input data via the HMD 104 or the HHC 106 by performingthe head motions and/or hand motions. Input data generated based on thehead motions and/or the hand motions is sent from the communicationscircuit of the HMD 104 or the HHC 106 to the computer 172. When theprocessor of the computer 172 receives the input data, the wirelessaccess application is executed to generate the list of wireless networksfor user selection to access the network 110.

The computer 172, in some embodiments, includes a network interfacecontroller (NIC) 174 that requests a portion of the game program 117from the game cloud 102. Examples of a NIC include a network interfacecard and a network adapter. The portion of the game program 117 isencoded by the codec 112 and streamed via the network 110 to the NIC 174of the computer 172. A processor 176 of the computer 172 executes theportion of the game program 117 to generate media data, which is sentfrom a communications circuit 178, e.g., transceiver, Transmit/Receivecircuit, a network interface controller, etc., of the computer 172, tothe HMD 104 for display on the display screen of the HMD 104. Acommunications circuit of the HMD 104 receives the media data from thecomputer 172 and sends the media data to the microcontroller of the HMD104 for processing and displaying the media data, including game scene,on the display screen of the HMD 104.

Moreover, the communications circuit 178 of the computer 172 receivesinput data generated based on the head motions from the HMD 104 and/orthe hand motions from the HHC 106 or actions performed at the HMD 104and sends the input data to the processor 176. The input data, in oneembodiment, may be real-world environment data captured by externalcamera 101 disposed on the outside face of the HMD 104 and transmittedby the communications circuit of the HMD 104. The processor 176 executesthe portion of the game program 117 b that is stored within the computer172 to generate additional media data, which is sent from thecommunications circuit 178 to the communications circuit of the HMD 104.Before or after receiving the additional media data, input data from theHMD 104 and/or the HHC 106 that is generated as part of the game playusing head motions and/or the hand motions, is sent by thecommunications circuit of the HMD 104 to the processor 176 via thecommunications circuit 178. In response to the input data, the processor176 executes the portion of the game program 117 b that is stored withinthe computer 172 to generate the next media data, which is sent from thecommunications circuit 178 to the communications circuit of the HMD 104.The next media data is sent to the communications circuit of the HMD 104to change the game play, including changing/updating virtual gameobjects and/or virtual environment of a game displayed by execution ofthe game program 117. When the game objects, e.g., real world objects,virtual game objects, etc., and/or virtual environment changes, a gamestate of the game displayed by execution of the game program 117changes.

In some embodiments, the game state is sent by the NIC 174 of thecomputer 172 via the router 152 and the network 110 to the game cloud102 to inform one or more servers of the game cloud of the current gamestate so as to synchronize the game state of the game on the game cloud102 with the game state on the computer 172. In such embodiments, mostof the game execution occurs on the computer 172.

In various embodiments, media data 116, additional media data, nextmedia data, etc., are initially sent from the codec 112 via the network110 and the router 152 to the HMD 104 until a portion of the gameprogram 117 is downloaded to the computer 172 from the game cloud 102.For example, initially, the user uses the game access application toaccess the game program 117. When the game program 117 is accessed, themedia data 116, the additional media data, the next media data, etc., issent from the codec 112 via the network 110 and the router 152 to theHMD 104 for display on the display screen of the HMD 104. During thetime of access of the media data from the game cloud 102 for display onthe HMD 104, the NIC 174 of the computer 172 downloads a portion of thegame program 117 from the game cloud 102 via the network 110 and therouter 152.

In some embodiments, when the game program 117 is accessed by thecomputer 172, media data, e.g., the media data 116, the additional mediadata, the next media data, etc., is sent from the codec 112 via thenetwork 110 directly to the HMD 104 for display on the display screen ofthe HMD 104 by bypassing the computer 172 while the computer accessesthe game program on the game cloud for downloading. The received mediadata is rendered on the display of the HMD 104. Meanwhile, the NIC 174of the computer 172 downloads a portion of the game program 117 from thegame cloud 102 via the network 110 and the router 152.

In a number of embodiments, a portion of input data generated based onthe head motions and/or hand motions and/or a portion of the real-worldenvironment data is sent from the HMD 104 via the router 152 and thenetwork 110 to the codec 112 of the game cloud 102 while the remainingportion of the input data and/or the remaining portion of the real-worldenvironment data is sent from the communications circuit of the HMD 104to the communications circuit 178 of the computer 172.

In various embodiments, a portion of input data generated based on thehand motions is sent from the communications circuit of the HHC 106 viathe router 152 and the network 110 to the codec 112 of the game cloud102 and the remaining portion of the input data is sent from thecommunications circuit of the HHC 106 to the communications circuit 178of the computer 172 either through the HMD or directly.

In various embodiments, media data, e.g., the media data 116, theadditional media data, the next media data, etc., that is generated byexecuting the game program 117 using the user input received from thecomputer/HMD/HHC, is sent from the codec 112 of the game cloud 102 viathe network 110 and the router 152 to the HMD 104 for rendering on thedisplay screen of the HMD 104 as part of game play and media data thatis generated by execution of the portion of the game program 117 by theprocessor 176 of the computer 172 is sent from the communicationscircuit 178 of the computer 172 to the HMD 104 for display on thedisplay screen. In these embodiments, the game cloud 102 and thecomputer 172 have synchronized game states. For example, the codec 112sends a game state generated by execution of the game program 117 viathe network 110 and the router 152 to the NIC 174 of the computer 172 toinform the computer 172 of the game state. As another example, the NIC174 of the computer 172 sends a game state generated by execution of theportion of game program 117 on the computer 172 via the router 152 andthe network 110 to the codec 112 of the game cloud 102 to inform the oneof more game cloud servers of the game state. The communication betweenthe codec 112 of the game cloud 102 and the NIC of the computer are doneperiodically to keep the game states synchronized on both sides.

In several embodiments, media data, e.g., the media data 116, theadditional media data, the next media data, etc., that is generated byexecuting the game program 117 and sent from the codec 112 of the gamecloud 102 to the HMD 104 has a higher amount of graphics than media datathat is generated by the processor 176 of the computer 172. As isevident, in some of the embodiments, the computer 172 is bypassed whenthe media data is directly sent from the codec 112 of the game cloud 102via the network 110 to the HMD 104.

In some embodiments, the computer 172 requests a portion of the gameprogram 117 from the game cloud 102 via the NIC 174 and in response, theportion 117 b of the game program 117 encoded by the codec 112 isstreamed via the network 110 to the NIC 174 of the computer 172. In someembodiments, the game cloud includes a games database 131 from which thegame program 117 is retrieved and downloaded to the computer 172. Insome embodiments, a portion 117 a of the game program 117 is downloadedfrom the games database 131 on to the game server 102 and the remainingportion 117 b of the game program 117 is downloaded to the computer 172.In some embodiments, the portion 117 b that is downloaded to thecomputer 172 is the entire game. The processor 176 of the computer 172executes the portion 117 b of the game program 117 to generate mediadata, additional media data and next media data (collectively termed‘media data’) which is sent from a communications circuit 178, a networkinterface controller, etc., of the computer 172, to the HMD 104 fordisplay on the display screen of the HMD 104.

As mentioned earlier, the additional media data and next media data maybe provided in response to input data, including head motions/other userinput, hand motions, etc., received from the HMD 104. In addition to thehead motions and/or hand motions the input data, in one embodiment, mayalso include real-world environment data that is captured by an externalcamera 101 disposed on the outside face of the HMD 104 and transmittedby the communications circuit of the HMD 104.

In some embodiments, the real-world environment data captured by theexternal camera 101 is stored locally within the HMD and used inrendering on the HMD screen. The additional media data provides virtualreality related data for rendering the virtual game scenes on the HMDand the next media data provides changes to virtual game objects and/orvirtual reality data displayed within the virtual game scenes duringgame play. A communications circuit of the HMD 104 receives the mediadata as a media stream from the computer 172 and sends the media data tothe microcontroller of the HMD 104 for interpretation and display on thedisplay screen of the HMD 104. When the game objects, e.g., real gameobjects, virtual game objects, etc., and/or virtual environment changes,a game state of the game displayed by execution of the game program 117,changes.

In some embodiments, a portion 117-a of the game program 117 is executedon the game cloud 102 while the game program 117 is being downloaded onto the computer 172. Accordingly, media data associated with theexecution of the portion 117-a of the game program 117 on the game cloud102, are sent directly from the codec 112 via the network 110 and therouter 152 to the HMD 104 for rendering on the HMD until the portion117-b of the game program 117 is downloaded to the computer 172 from thegame cloud 102. In one embodiment, the portion 117-b of the game program117 is downloaded and stored in the local storage 113 of the computer172 and executed by the processor 176. Once the portion 117-b isdownloaded and the processor 176 of the computer 172 starts executingthe game portion 117-b, the media data will be transmitted from thecomputer 172 to the HMD 104 for the portion 117-b of the game program117. In some embodiments, all the media data for the game program aretransmitted directly from the computer 172 to the HMD 104 for rendering.The computer 172 may also periodically transmit the media data to thegame cloud 102 to synchronize the game state of the game program on thegame cloud 102 with the game state on the computer 172.

In a number of embodiments, a portion of input data based on the headmotions and/or hand motions are captured by an observation camera (notshown) that is connected to the computer 172. in some embodiments, theconnection between the observation camera and the computer 172 may be awired connection. In other embodiments, the connection between theobservation camera and the computer 172 may be a wireless connection. Insome embodiments, the observation camera is any one or combination ofstereo camera, IR camera or mono-camera. In some embodiments theobservation camera is one of a video camera or a still-motion camera.The images captured by the observation camera may be used to determinethe location and motion of the HMD and the HHC. For example, the imagesof the observation camera may be used to identify coordinates of aposition of the HMD and coordinates of a position of the HHC. Inaddition to the coordinates of the coordinate plane, the images of theobservation camera may be used to determine the pitch, the yaw and theroll to generate the six-axis data for the HMD and HHC. In someembodiments, the head and/or hand motions generated at the HMD and theHHC are captured by the observation camera and transmitted to themicrocontroller of the HMD 104 as six axis data. The six-axis data fromthe HMD 104 and/or HHC 106 are interpreted to generate the input data.The interpreted input data is transmitted from the HMD 104 to thecomputer 172 to influence the outcome of the game program. In someembodiments, the head and/or hand motions captured by the observationcamera are directly transmitted to the processor 176 where it isinterpreted to generate the six-axis data. The observation cameraobserves the motions (head and/or hand) of the user and this informationis used in providing feedback to the game program to influence the gamestate changes. In this embodiment, any other input data related to thegame program 117 are transmitted by the HMD 104 to the processor and theprocessor 176 interprets the other input data with the six-axis data todetermine if the game state of the game needs to be altered. Based onthe interpretation, the game state of the game is changed. In someembodiments, the input data from the HMD 104 includes real-worldenvironment data captured by the external camera 101 and sent from thecommunications circuit of the HMD 104 to the communications circuit 178of the computer 172. The real-world environment data may be used toinfluence the virtual game scenes rendered at certain portions of thescreen of the HMD 104. In various embodiments, the HMD 104 is used todisplay a two-dimensional or a three-dimensional image.

In some embodiments, the observation camera may be a video camera thattracks the motions provided at the HMD and the HHC. The observationcamera captures image of various marker elements disposed on the faceplate of the HMD and the HHC, and correlates the position of the markerelements of the HMD and the HHC to a three-dimensional space. Eachmarker element may be a light emitting diode, an infrared light, acolor, a reflective material, an object with special features orcharacteristics that are easily recognized via image analysis, etc. Inaddition, the HMD 104 may also include special visual markers (notshown), such as reflective areas of particular geometrical shape, areaswith a particular color (e.g., blue rectangle, etc.), or markings (e.g.,three parallel lines on the surface of the HMD). In some embodiments,the HMD also includes additional marker elements on the side and/or backof the HMD (i.e., the part of the HMD touching the back of the head) tofurther visually track the location of the HMD by detecting therespective lights or visual markers.

In various embodiments, the observation camera may be a mobile videocamera. For example, the video camera (not shown) may be attached to arobotic device, e.g., a multicopter, a robotic arm, a robot, a roboticvehicle, a robotic car, a quadcopter, etc. For example, the video cameramay be attached under, on top of, to a side of the robotic device forcapturing images of the HMD and/or the HHC. The HMD moves with movementof a head of the user 108. In several embodiments, instead of the videocamera, a digital camera may be used.

In some embodiments, the observation camera may be a stereo camera,which is a camera that includes two or more lenses with separate imagesensor for each lens. The separate image sensor enables the stereocamera to capture three-dimensional images of an object that provide anillusion of depth.

In another embodiment, the observation camera may be an infrared (IR)camera that is used to analyze infrared light provided on the HMD. Theinfrared light is not visible to the human eye but can be easilydetected by the infrared camera. The HMD may include infrared lights toavoid distraction in the appearance of the HMD. In some environments(e.g., low light or bright light), it may be easier to track infraredlight than other types of lights for detecting location, shape and orfeatures in the HMD. The infrared (IR) cameras provide enhanced imagingand thermal imaging of a tracking object, such as the HMD. The IRcameras may also be used as gaze detection cameras to detect user's gazedirection.

In yet another embodiment, the observation camera may be a regularcamera. In some embodiments, the observation camera may be a monocamera, wherein the lens (i.e., single lens), is used to track thelights or other marker elements in the HMD that are configured forvisual tracking. In order to determine the depth of the HMD within thefield of play with the regular camera, the size of some of the featureson the HMD are analyzed. The smaller the features are, the further awaythe features are supposed to be from the camera of the HMD. In someembodiments, the depth of the HMD within the field of play may bedetermined by using more than one observation camera. In addition, thevisual tracking may also be combined with other types of tracking, suchas inertial motion tracking, dead reckoning, ultrasound communicationbetween the HMD and the computing device, etc.

The observation camera captures an image of the HMD 104 by tracking theone or more marker elements of the HMD. When the head of the user 108tilts or moves, position and location of the marker elements of the HMDchanges in a coordinate system. The digital camera captures an image ofthe marker elements and sends the image to the computer 172. An image ofthe marker elements is an example of input data. Position of the HMD 104in a three dimensional space (X, Y, Z) can be determined by theprocessor 176 of the computer 172 based on the positions of the markerelements in the images. Further, inertial motion, e.g., yaw, pitch, androll, etc., of the HMD 104 is determined by the processor 176 of thecomputer 172 based on movement of the marker elements. In the caseswhere the computer 172 is not available, the image of the markerelements from the observation camera are sent to the processor of theHMD 104 and the HMD's processor will determine the position of the HMDusing the coordinates of the marker elements.

In some embodiments, the observation camera captures an image of the HHC106. When the hand of the user 108 tilts or moves, position and locationof the marker elements on the HHC changes in a coordinate system. Theobservation camera captures an image of the marker elements on the HHCand sends the image to the computer 172 or to the processor of the HMD104. An image of the marker elements on the HHC is an example of inputdata. Position of the HHC 106 in a three dimensional space (X, Y, Z) canbe determined by the processor 176 of the computer 172 or by theprocessor of the HMD 104 by analyzing the positions of the markerelements on the HHC in the image. Moreover, inertial motion, e.g., yaw,pitch, and roll, etc., of the HMD 104 is determined by the processor 176of the computer 172 or the processor of the HMD 104 based on movement ofthe marker elements of the HHC.

In some embodiments wherein the HMD 104 is communicatively connected tothe computer 172 using a wired connection, the HMD is configured todetect a break in the wired connection so as to pause the virtual gamescenes rendered on the screen of the HMD 104. The HMD detects a break inthe communication connection, generates a signal accordingly and relaysthe signal to the computer 172 to cause the computer 172 to pause theexecution of the game program and to store the game state and gamescenes for the session for the game. Power from a battery of the HMD maybe used to provide the power for communicating with the computer 172during the break in the communication connection, the status of theconnection. The execution of the game program may resume as soon as thecomputer 172 gets a signal from the HMD 104 that the wired connectionhas been re-established. In some embodiments, upon resumption of theconnection between the HMD and the computer 172, the computer 172 maystart streaming the game scenes from the point of disruption. In anotherembodiment, the computer 172 may start streaming the game scenes from apoint before the pause (for example, few hundred frames before thepause) caused by the connection disruption so that the user may get sometime to immerse in the game. In this embodiment, the computer 172 mayallow the user to re-execute portions of the game to allow the user toget into the game. The communication between the HHC and the HMD and thecommunication between the HHC and the computer 172 may follow a wirelesscommunication protocol.

In some embodiments, the HMD 104 may include one or more internalcameras (e.g., gaze detection cameras) 103 to detect changes in theuser's eyes movement, gaze direction, gaze pattern, etc. The internalcameras 103 may also be used to identify/authenticate the user beforeproviding access to the game.

Although detailed description is provided regarding a gamingenvironment, it is envisioned that the interfacing can also take placeduring interactive communication with a computer system. The computersystem can be a general computer, with a graphical user interface thatallows user to present and make gestures in space, that control icons,entry, selection, text, and other commands.

For more information regarding the method for following a marked object,reference may be made to U.S. Patent Application Publication No.2012-0072119, filed on Aug. 15, 2011 and published on Mar. 22, 2012, andU.S. Patent Application Publication No. 2010-0105475, filed on Oct. 27,2008 and published on Apr. 29, 2010, both of which are hereinincorporated by reference in its entirety.

In some embodiments, one or more pairs of stereo camera, one or moreinfrared cameras and/or one or more regular camera or combinationsthereof may be used to determine the relative position of the HMD andthe motion of the HMD provided by user's head motion as well as thecontroller, including the user's hand wearing a wearable article/devicethat is used to provide input data.

The one or more internal cameras (e.g., gaze detection cameras, etc.)may be mounted on the HMD and facing inward toward the user to captureimages related to the user and feed the images to the communicationmodule to provide user specific and environment specific data to theHMD. The internal camera(s) may be used to identify a user wearing theHMD, which can be used to obtain user profile of the user. Accordingly,the internal cameras may be configured to engage retinal scanningtechnique and/or iris scanning technique to scan the user's retina oriris and use the data from the scanning to generate at least onebiometric identity of the user. The user's biometric identity may bepart of the user's profile. The internal cameras may also include a gazedetection camera that are equipped with gaze detector algorithm todetect the direction of the user's gaze and to adjust the image datarendered on a screen of the HMD based on the detection. In someembodiments, the internal cameras are IR cameras. The gaze detectiontechnology may also be used to authenticate a user. For example, theuser may be asked to follow an object rendered on the screen or track arandomly generated letter, object or pattern (for e.g., a circle, atriangle, a rectangle, etc.) that is rendered on the screen. In someembodiments, verbal or textual commands may be provided for a user totrack a letter, an object or pattern on the screen and the userauthenticated by using the gaze detection technology. The authenticationof a user may be used to allow access to a user account, to a game, tocertain parts or levels of a game, etc.

The internal cameras and the external cameras of the HMD workhand-in-hand to determine the gaze of the user and to relate the gaze toan object in the line-of-sight of the user's gaze. The game processingmodule of the HMD includes the software to compute the direction of theuser's gaze and correlate it to objects within the field of view of thecomputed direction.

For example, the internal cameras 109 detect and track the user's eyemovement and gaze. The internal cameras 109 may be used to determine theuser's gaze direction for a period of time (e.g., when the user islooking at a particular object or point in the images rendered on thedisplay screen, for some period of time), detect a gaze pattern over aperiod of time (for e.g., when a user follows an object, traces apattern, etc.), and/or detect changes in gaze directions (for e.g.,back-and-forth movement of the eyes, rolling of the eyes—which may be asign of the user experiencing dizziness or fatigue—especially in a highintensity game, etc.). The HMD's internal cameras communicate with theoutside mounted cameras of the HMD and with the observation cameras todetermine if data provided for rendering on the screen of the HMD needsto be adjusted, in response to detected eye movement or gaze or based ontriggered events occurring within the game or in the environment in theimmediate vicinity of the user wearing the HMD.

FIG. 2 is a block diagram of a communication architecture of an HMD 104.The HMD 104 includes some exemplary control modules or sensors, such asa video audio separator 254, a video decoder 255, a memory device 256, aWAC 258, a stream buffer 259, one or more speakers 260, a battery 261, auser input circuit 262, a display screen 266, a microcontroller 268, anaudio buffer 272, an observation digital camera 274, an external digitalcamera 274, an audio codec 276, an internal digital camera 278, a videobuffer 280, a video audio synchronizer 282, a microphone 284, LEDs 285and IR lights 287, a controller/computer communications circuit 289. TheLEDs 285 and IR lights 287 represent the marker elements that are usedto track the position of the HMD.

In a number of embodiments, the speakers 260 form an audio circuit. Invarious embodiments, the audio codec 276, the audio buffer 272, and/orthe speakers 260 form an audio circuit. In various embodiments, themicrocontroller 268 is part of a display circuit that controls imagesrendered on a display screen. Examples of a display screen 266 includean LED screen, a liquid crystal display (LCD) screen, a liquid crystalon silicon (LCoS) screen, an organic LED (OLED) screen, a plasma screen,etc. An example of the external digital camera includes an eye camera,such as Playstation Eye® manufactured by Sony Computer Entertainment,Inc.

The microcontroller 268 stores a rendering program 286 and an operatingsystem 288. The rendering program 286 and the operating system 288 arestored in a memory device of the microcontroller 286 and executed by amicroprocessor of the microcontroller 268. An example of microcontroller268 includes a low cost microcontroller that includes a driver, e.g., anLCD driver, that generates a signal to detect elements (for e.g., LCDs,etc.), to provide media data, for displaying on the display screen 266.Another example of the microcontroller includes a GPU and a memorydevice.

In some embodiments, the memory device of the microcontroller is otherthan a flash memory or a random access memory (RAM). For example, memorydevice of the microcontroller is a buffer. In various embodiments,memory device of the microcontroller is a flash memory or a RAM.Examples of the user input circuit 262 include a gyroscope, amagnetometer, and an accelerometer. In some embodiments, the user inputcircuit 262 also includes a global position system (GPS), compass or anylocation tracking devices. An example of the WAC 258 includes a NIC. Insome embodiments, the WAC 258 is referred to herein as a communicationscircuit.

A stream of encoded media data is received into the stream buffer 259from the network 110 or the router 152. It should be noted that when therouter 152 is coupled to the computer 172, data received from thecomputer 172 is stored in a buffer (not shown) of the HMD 250 or in thememory device 256 instead of being stored in the stream buffer 259.

The WAC 258 accesses the stream of encoded media data from the streambuffer 259 received from the computer or the codec 112 and de-packetizesthe stream. The WAC 258 also includes a decoder to decode the encodedmedia data.

In embodiments in which the stream of encoded media data is received bythe computer 172 via the router 152, the NIC 174 of the computer 172de-packetizes and decodes the stream of encoded media data to generatedecoded data, which is stored in the buffer (not shown) of the HMD 250.

The decoded data is accessed by the video audio separator 254 from theWAC 258 or from the buffer (not shown). The video audio separator 254separates audio data within the decoded data from video data.

The video audio separator 254 sends the audio data to the audio buffer272 and the video data to the video buffer 280. The video decoder 255decodes, e.g., the video data and/or changes to the video data from adigital form to an analog form to generate analog video signals. Thevideo audio synchronizer 282 synchronizes the video data stored in thevideo buffer 280 with the audio data stored in the audio buffer 272. Forexample, the video audio synchronizer 282 uses a time of playback of thevideo data and the audio data to synchronize the video data with theaudio data.

The audio codec 276 converts the synchronized audio data from a digitalformat into an analog format to generate audio signals and the audiosignals are played back by the speakers 260 to generate sound. Themicrocontroller 268 executes the rendering program 286 to display a gameon the display screen 266 based on the analog video signals that aregenerated by the video decoder 255. In some embodiments, the gamedisplayed on the display screen 266 is displayed synchronous with theplayback of the audio signals.

Moreover, the user speaks into the microphone 284, which converts soundsignals to electrical signals, e.g., audio signals. The audio codec 276converts the audio signals from an analog format to a digital format togenerate audio data, which is stored in the audio buffer 272. The audiodata stored in the audio buffer 272 is an example of input datagenerated based on a sound of the user. The audio data may also includeother audio signals generated at the HMD or detected by the speakers inthe HMD. The audio data is accessed by the WAC 258 from the audio buffer272 to send via the network 110 to the codec 112 of the game cloud 102.For example, the WAC 258 packetizes and encodes the audio data accessedfrom the audio buffer 272 to send via the network 110 to the codec 112.

In some embodiments, the audio data is accessed by the WAC 258 from theaudio buffer 272 to send via the router 152 and the network 110 to thecodec 112 of the game cloud 102. For example, the WAC 258 packetizes andencodes the audio data accessed from the audio buffer 272 to send viathe router 152 and the network 110 to the codec 112.

The internal digital camera 278 captures one or more images of the eyemotions of the user wearing the HMD to generate image data, which is anexample of input data generated at the HMD. based on the head actionsand/or eye movements. Similarly, the observation digital camera 274and/or the external digital camera 274 mounted on the HMD captures oneor more images of the hand of the user 108, and/or of the markerslocated on the HMD 250 and/or on the HHC/glove/hand of the user 108,head motions of the user wearing the HMD, to generate image data, whichis an example of input data that is generated based on the hand/headmotions. The image data captured by the digital cameras 274, 275 and 278is stored in the video buffer 280.

In some embodiments, the image data captured by the digital cameras 274,275 and 278 is stored in a buffer of the HMD 250 and the buffer is otherthan the video buffer 280. In various embodiments, the image datacaptured by the digital cameras 274, 275 and 278 is decoded by the videodecoder 255 and sent to the microcontroller 268 for display of images onthe display screen 266.

The image data captured by the digital cameras 274, 275 and 278 isaccessed by the WAC (wireless access card) 258 from the video buffer 280to send via the network 110 to the codec 112 of the game cloud 102. Forexample, the WAC 258 packetizes and encodes the image data accessed fromthe video buffer 280 to send via the network 110 to the codec 112.

In some embodiments, the video data is accessed by the WAC 258 from thevideo buffer 280 to send via the router 152 and the network 110 to thecodec 112 of the game cloud 102. For example, the WAC 258 packetizes andencodes the video data accessed from the video buffer 280 to send viathe router 152 and/or the network 110 to the codec 112.

The controller/console communications circuit 289 receives media datafrom the computer 172 for storage in the buffer (not shown). Moreover,the controller/console communications circuit 289 receives input signalsfrom the HHC 106, converts the input signals from an analog form to adigital form to generate input data, which is accessed by the WAC 258 tosend via the network 110 to the codec 112 of the game cloud 102. Forexample, the WAC 258 packetizes and encodes the input data accessed fromthe controller/console communications circuit 289 to send via thenetwork 110 to the codec 112.

In some embodiments, the input data is accessed by the WAC 258 from thecontroller/console communications circuit 289 to send via the router 152and the network 110 to the codec 112 of the game cloud 102. For example,the WAC 258 packetizes and encodes the video data accessed from thevideo buffer 280 to send via the router 152 and the network 110 to thecodec 112.

It should be noted that instead of the controller/console communicationscircuit 289, two separate communications circuits may be used, one forcommunicating, e.g., receiving, sending, etc., data with the computer172 and another for communicating data with the HHC 106.

In a number of embodiments, the decoder is located outside the WAC 258.In various embodiments, the stream buffer 259 is located within the WAC258.

In some embodiments, the HMD 104 excludes the observation digital camera274. In several embodiments, the HMD 104 includes any number ofmicrocontrollers, any number of buffers, and/or any number of memorydevices.

In various embodiments, the HMD 104 includes one or more batteries 261that provide power to components, e.g., the video audio separator 254,the memory device 256, the wireless access card 258, the stream buffer259, the one or more speakers 260, the user input circuit 262, thedisplay screen 266 the microcontroller 268, the audio buffer 272, theexternal digital camera 274, the audio codec 276, the internal digitalcamera 278, the video buffer 280, the video audio synchronizer 282, themicrophone 284, and the controller/computer communications circuit 289.The one or more batteries 261 are charged with a charger (not shown)that can be plugged into an alternating current outlet.

In a number of embodiments, input data and/or media data is referred toherein as interactive media.

In some embodiments, the HMD 104 includes a communications circuit tofacilitate peer-to-peer multichannel communication between local usersvia pairing. For example, the HMD 104 includes a transceiver thatmodulates sound signals received from the microphone 284 and sends themodulated signals via a channel to a transceiver of another HMD (notshown). The transceiver of the other HMD demodulate the signals toprovide to speakers of the other HMD to facilitate communication betweenthe users.

In various embodiments, different channels are used by the transceiverof the HMD 104 to communicate with different other HMDs. For example, achannel over which the modulated signals are sent to a first other HMDis different than a channel over which modulated signals are sent to asecond other HMD.

In some embodiments, the WAC 258, the user input circuit 262, themicrocontroller 268 and the video decoder 255 are integrated in one ormore individual circuit chips. For example, the WAC 258, the videodecoder 255 and the microcontroller 268 are integrated in one circuitchip and the user input circuit 262 is integrated into another circuitchip. As another example, each of the WAC 258, the user input circuit262, the microcontroller 268 and the video decoder 255 is integrated ina separate circuit chip.

The various modules of the HMD are used to detect user's gaze directionand/or actions at the HMD and adjust the images presented on the displayscreen of the HMD to correspond with the detected gaze direction and/oractions.

FIG. 3 illustrates a broad overview of the various communication modulesthat are involved within the HMD and the computer to detect gazedirection and/or actions provided by the user and to adjust the imagespresented at the display screen of the HMD, in some embodiments. Itshould be noted that only a few of the modules are shown to be involvedin adjusting the images that are being presented at the display screenof the HMD, but in reality other modules within the HMD are involved inthe process of adjusting the images presented on the HMD. Theembodiments may be used to provide augmented reality of the real-worldenvironment by enhancing a portion of the images of the real-worldenvironment captured by one or more forward facing cameras mounted onthe HMD. Alternately, the embodiments may be used to adjust virtualreality (VR) scene presented at the HMD. The VR scene may be from apre-recorded video of a game play of a user, a pre-recorded video of aplace or event, a game scene of a video game, etc.

In one embodiment, the HMD is used to augment real-world environment. Inthis embodiment, the various components of the HMD 104 work withcomponents of the computer 172 to detect a gaze direction of the userand to enhance certain portion(s) of the images rendered on the screenof the HMD that is in line with the detected gaze direction of the user.

In this embodiment, the controller/console communication circuit (orsimply referred to herein as “CC communication circuit”) 289 of the HMD104 receives images of a real-world environment captured from a vicinityof the user wearing the HMD 104 by one or more forward facing cameras274, as image frames. The image frames are processed by the CCcommunication circuit 289. As part of the processing, the image framesmay be encrypted in accordance to communication protocol established forthe communication connection between the HMD and the computer 172, andstreamed to the computer 172.

The communication device 178 of the computer 172 receives the streamingimage frames and decrypts the data and forwards the decrypted image datato the processor. The processor includes an image analyzing module (notshown) that is configured to analyze the image data by constructing athree dimensional model of the real-world environment captured in theimages using different modeling tools that are well-known in theindustry (e.g., using multiple stereoscopic cameras). The threedimensional model is maintained in memory and updated as and when newimage data is received from the HMD. In some embodiments, the processormay generate a two-dimensional format of the images and forward the sameto the HMD for rendering on the display screen of the HMD. For moreinformation on converting image data from the three-dimensional formatto a two-dimensional format, reference can be made to application Ser.No. 14/220,420, filed on Mar. 20, 2014, and entitled, “SharingThree-Dimensional Gameplay,” which is incorporated herein by referencein its entirety. During the viewing of the real-world environment,images from a gaze detection camera may be simultaneously received,processed by the CC communication circuit and forwarded to the computeras gaze image frames. The image analyzing module analyzes the gaze imageframes to identify the gaze direction of the user. When it is determinedthat the user's gaze is directed to a particular portion of the displayscreen of the HMD, the image analyzing module may correlate the detectedgaze direction of the user to an object or point within the generatedmodel of the real-world environment to identify an object or point thathas captured the user's attention.

The image analyzing module continues to track the gaze direction of theuser over the duration of viewing of the real-world environment. When itis determined that the user's gaze has been directed to the object orpoint for a pre-defined period of time, the image analyzing module maygenerate a signal instructing the HMD to adjust a zoom factor for lensof the one or more forward facing cameras so that the object or point ofinterest to the user can be zoomed in. In some embodiments, the signalto adjust the zoom factor may include a signal to adjust a focal lengthof the lens of the forward facing cameras so as to cause the camera tozoom in on the object. In some embodiments, the signal to adjust thefocal length may include signal to control speed of adjusting the focallength. The speed of adjustment of the focal length may be driven by thetype of content that is being captured in the real-world environment ormay depend on the user. For example, if the user is able to handle afaster zoom in, the speed of adjusting the focal length may be set highas opposed to a user who cannot handle faster zoom in. In someembodiments, the signal to adjust the zoom factor may also include asignal to adjust an aperture setting of lens of the one or more forwardfacing cameras so as to affect the depth at which the image of theobject is captured. In addition to providing a signal to adjust thefocal length and speed, the signal may also include a signal to adjustbrightness level of the screen (i.e., screen of the HMD or an externaldisplay surface) so that the object rendered on the screen appear clearand in focus.

In some embodiments, the signal may be generated to enhance the audiocomponent captured in the image frames of the real-world environment. Insuch embodiments, the signal may be generated to adjust only the audiocomponent. For example, the signal to adjust the audio component may begenerated when it is determined that the user wearing the HMD isapproaching an edge of a VR scene or a boundary of an interactive regiondefined for the user in the physical environment or is getting proximalto an object or person in the physical environment.

In alternate embodiments, only the video component may be enhanced. Insome embodiments, a command to enhance the audio, video or both theaudio and video components of the image frames may be provided by auser. In such embodiments, the CC communication circuit may receive andforward the command from the user to the computer so that the imageanalyzing module in the computer can identify the appropriate componentor portion of the real-world images that need to be enhanced. In someembodiment, the command may be provided through audio commands and suchaudio commands may be received through the microphone controller 284,processed by the CC communication circuit 289 and forwarded to thecomputer for further processing. The computer processes the commands andgenerates appropriate signal to the HMD so that the relevant contentrendering components may be adjusted to provide an enhanced content.

FIGS. 4A-4C illustrate an example of adjusting an image of a real-worldenvironment that is rendered at the HMD, in response to detecting a gazedirection of a user, in one embodiment. FIG. 4A illustrates a userviewing a real-world lake scene captured by the forward facing camerasof the HMD and rendered on a display screen of the HMD, at time t0.While rendering the lake scene, the gaze detection cameras keep track ofthe user's gaze direction to determine if the user's gaze is fixed on apoint or object rendered on the display screen. In one embodiment,images captured by gaze detection camera(s) is analyzed to identify thatthe user's gaze direction is indeed directed toward a particular portionor point of the display screen. It is then determined if the user's gazedirection toward the particular portion or point of the display screenlasts for at least a pre-defined period of time. If it is determinedthat the user's gaze has shifted, no trigger event is initiated by theHMD and the HMD continues to render the images from the lake scene. If,however, it is determined that the user's gaze continues to be directedtoward the particular portion or point of the display screen for atleast the pre-defined period of time (e.g., 4 seconds, 6 seconds, etc.),a trigger event is initiated at the HMD. As part of initiating thetrigger event, images from the gaze detection cameras are forwarded asgaze detection image frames to the computer, for processing, in oneembodiment. In another embodiment, the coordinates of the particularportion or point of the display screen is computed at the HMD andforwarded to the computer as user gaze data for further processing.

In the embodiment where gaze detection image frames are provided, animage analyzing module available at the computer is used to analyze thegaze detection image frames and correlate the user's gaze direction to aparticular object or point in the lake scene by mapping the gazedirection to a three-dimensional model of the lake scene generated bythe image analyzing module. As shown in FIG. 4B, based on the analysis,it is determined that the user's gaze direction corresponds to a messageboard that is at the end of a pier within the lake scene. Further, theanalysis identifies that the object (i.e., message board) is rendered ata virtual distance D1 on the display screen making the message hard forthe user to read.

In the embodiment where the user gaze data identifying coordinates isprovided to identify the particular portion or point of the displayscreen, the image analyzing module available at the computer correlatesthe coordinates provided in the user gaze data to the particular objector point in the lake scene. In some embodiments, the coordinates areprovided in three-dimensional format and the image analyzing module maycorrelate the coordinates to a three-dimensional model generated for thelake scene. In some embodiments, the coordinates are provided intwo-dimensional format and the image analyzing module may correlate thecoordinates to a two-dimensional model generated for the lake scene.

Upon identifying the particular point or object of interest, thecomputer forwards a signal to the HMD to adjust the zoom factor of thelens of the forward facing cameras so that the object captured in thelake scene is zoomed in, as illustrated in FIG. 4C. In response to thesignal, the object is now presented at a virtual distance D2, wherein D2is less than D1, thereby making the message board bigger so that theuser can read the warning message. The speed of zooming is adjusted toensure that the zooming in does not cause any discomfort ordisorientation to the user. Although the various embodiments discussenhancing a portion of the image component of the real-world scenecaptured by the forward facing cameras, the embodiments may be extendedto enhance the audio portion or the video portion captured in thereal-world environment as well. In some embodiments, the signal toadjust the zoom factor may include a signal to adjust the focal lengthof the lens of the forward facing cameras. In other embodiments, inaddition to or instead of the signal to adjust the focal length, asignal to adjust the aperture of the lens may be included so that theobject or point that is enhanced appears sharp. This may entailadjusting the lens aperture to ensure sufficient light passes through tothe image sensor to provide a clear and enhanced view of the object thathas captured the user's interest. The signal may also specify a speed ofsuch adjustments so as to provide the user with an enhanced image of theobject without subjecting the user to any discomfort or causingdisorientation or dizziness or motion sickness.

In some embodiments, the HMD and the computer may be used to enhanceportions of a virtual reality scene, such as game scene of a video game,currently rendering on the display screen of the HMD. In suchembodiments, the controller/console communication circuit (or simplyreferred to herein as “CC communication circuit”) 289 of the HMD 104receives the game data from the computer 172 in media streams, forwardsthe media streams to the image processing module 290, where the mediastreams are decrypted to generate the game data. The image processingmodule 290 engages other components of the HMD to process differentcomponents of the game data. The processed game data is forwarded tocorresponding components of the HMD. For example, the image processingmodule 290 may engage a video/audio separator 254 to separate the videoand the audio component of the game data. The image processing module290 then engages a video decoder 255 (FIG. 2) to decode the videocomponent of the game data to identify video frames and forward thevideo frames to the display controller for rendering on the displayscreen. The video decoder 255 may use a video buffer 280 to providebuffering of the video component of the game data before it is forwardedto the display controller (i.e., display screen 266) for rendering. Thevideo buffering is to minimize latency for the game data presented onthe display screen 266. Similarly, the image processing module 290 mayengage an audio decoder 276 (FIG. 2) to decode the audio component ofthe game data prior to rendering the audio data on the one or morespeakers of the HMD. The audio decoder 276 may use an audio buffer 272to provide buffering for the audio component of the game data prior toforwarding the audio portion to the speakers of the HMD. Of course, theaudio component and the video component are synchronized when presentedat the respective components of the HMD.

In some embodiments, during the presentation of the game data, theuser's motions and actions are tracked at the HMD to determine if atrigger event needs to be initiated. For example, a trigger event may beinitiated at the HMD when it is determined that the user's gaze isdirected to a particular object or point on the display screen for atleast a pre-defined period of time (e.g., 2 seconds, 5 seconds, etc.).One or more gaze detection camera 278 is engaged to monitor the user'sgaze in order to determine the gaze direction of the user. The gazedetection camera tracks the user's gaze and determines the gazedirection by mapping the user's gaze to corresponding portion of contentrendered on the display screen of the HMD. When it is determined thatthe user's gaze has been directed to a particular point, portion orobject of content rendered on the screen for a pre-defined period oftime, the gaze detection camera captures the images of the user's eyesand forwards the images in image frames to the CC communications circuit289 for processing. The CC communications circuit 289 analyzes theinputs to detect the gaze direction of the user, initiates a triggerevent when it determines that the user's gaze is fixed on a particularpoint or object of content for at least a pre-defined period of time,and forwards data related to user's gaze (e.g., coordinates, images,etc.) to the communication device 178 of the computer 172. The computer172 uses the gaze direction data and maps the data to the video imagesthat are currently being provided for rendering on the screen of the HMDto identify the particular point or object of the game data that hascaptured the user's interest.

In some embodiments, the user's gaze direction may be influenced by avisual cue provided with or in the game data or in the real-worldenvironment. For example, the visual cue may be in the form of a beaconor a directional arrow or a highlight or color flash, etc., at or nearan object or point of interest. The visual cue may capture the user'sattention leading to a shift in the user's gaze direction toward theobject that is rendered at a virtual distance. Alternately, the user'sgaze direction may be influenced by change in the user's gazecharacteristics, such as narrowing of the user's eyes trying to focus ona particular object that appears at a virtual distance, when rendered onthe display screen of the HMD 104. The object or point of interest, forexample, may be hard for a user to see clearly. In some embodiments, inresponse to the initiated trigger event, the computer 172 may generate asignal instructing the game logic to adjust the game data so that theobject is brought into focus. In alternate embodiments, the signal mayprovide instructions to the HMD to adjust the rendering of the images soas to digitally zoom in on the object or point of interest in the VRscene.

In some other embodiments, in response to the occurrence of the triggerevent caused by gaze detection, the signal generated by the computer mayinclude instructions to adjust optical settings of the lens of the HMD.For example, the signal may include instructions to adjust a zoom factorfor the lens of the HMD so that the object rendered on the screen may bezoomed in and brought into focus. The adjustment to the zoom factorcauses a change in the virtual distance at which the object is renderedon the display screen of the HMD from a first virtual distance to asecond virtual distance, wherein the second virtual distance is shorterthan the first virtual distance. The adjustment in the virtual distancemakes it appear that the object is brought closer to the relativeposition of the user within the VR scene.

In some embodiments, the VR scene may include one or more points orobjects that may be of interest to users. As mentioned earlier, the VRscene may be a pre-recorded video provided by a user or a contentprovider, a recording of a video game play of a user, etc. Thepre-recorded video may include one or more objects or points of interestthat may have been identified by other users or content provider asbeing interesting. The objects may be identified using a tag, a beacon,a highlight, color flash, or any other form of visual or audio cue.These visual cues in the VR scene may influence the user to gaze in aparticular direction. The image analyzing module in the computerdetermines the gaze direction of the user and identifies one of theobjects or points within the VR scene that have been previouslyidentified and is in line with the gaze direction.

In some embodiments, once the object or point of interest has beenidentified, the user may be provided with an option to confirm theidentity of the object or point of interest. For example, the option maybe provided as a button on the HMD or the HHC that is communicativelycoupled to the HMD for user selection, an option presented on an userinterface at the display screen of the HMD that can be selected usingcontrols provided in the HHC or the HMD, an audio option, a gestureoption (e.g., a nod or a wave captured by an external image capturingdevice, a wink or a blink provided by the user and detected by an imagecapturing device of the HMD, a tap on an input pad of the HHC or HMD, aswipe action on an input pad of the HHC or HMD, etc.). User selectionand/or action provided at the options are captured using the user inputcircuitry 262 and processed by the CC communication circuit 289. Basedon the user selection, a signal may be generated by the computer to theHMD to adjust the zoom factor of the lens of the HMD or to adjust theimage of the VR scene so that the selected object may be brought intofocus at a virtual distance that makes the object appear clear, sharpand close to the user's eyes to make it discernible.

In some embodiments, an image processing algorithm available at thecomputer may use the points of interest identified in the VR scene byvarious users, and define areas that are proximal to the points ofinterest where a user can be teleported. The areas are pre-defined so asto take into account the terrain attributes near or around the points ofinterest. For example, if a mountain scene is being presented on thedisplay screen of the HMD, the various points of interest may beidentified at different elevations of the mountain as potential vistapoints. These points of interest are used to define virtual areas forteleporting a user, in some embodiments. When the virtual areas aroundor proximal to objects or points of interest are defined, the imageprocessing algorithm takes into consideration the terrain or otherfeatures of the VR scene. In some embodiments, the pre-defined virtualareas near or around the objects or points of interest in the VR scenemay be used to “teleport” the user. These virtual areas are mapped to aphysical space in which the user wearing the HMD is operating and definea zone of movement within the physical space for the user to provideinteractions while viewing the VR scene.

Teleporting the user, in one embodiment, may include adjusting the imageof the VR scene presented on the display screen of the HMD. Theadjusting of the image of the VR scene is done in such a manner so as tomake it appear that the user has been transferred to a new locationdefined by the pre-defined virtual area associated with the object orpoint in the VR scene that has caught the user's interest or attention,as defined, for example, by the gaze direction. In some embodiments, aspart of teleporting the user, the image of the VR scene is adjusted sothat the VR scene that is rendered at the HMD reflects the view from thepre-defined virtual area. For example, in the mountain scene, as theuser is teleported to a pre-defined area that is close to a vista point,the content that is rendered on the display screen of the HMD is a viewof the mountain scene as seen from the vista point. After rendering theadjusted content, any movement by the user in the physical space istranslated to the user's movement in the teleported pre-defined virtualarea at the vista point and the content of the virtual scene is adjustedin accordance to the detected user's movement. For example, if the usermoves his head to the right, the scene from the right side of thevirtual scene is rendered and when the user moves his head down, thevalley scene in line with the user's gaze direction is rendered. In someembodiments, the adjustment to the VR scene is performed for apre-defined period of time and after expiration of the pre-definedperiod of time, the image of the VR scene before the adjustment isrendered.

FIGS. 5A and 5B identify one such example of a VR scene that identifiesvarious vista points or objects/points of interest identified by otherusers or by a computer to which a user may be teleported, in oneembodiment. FIG. 5A illustrates a scene of a rest stop along a scenicroute that is rendered on the display screen of the HMD of the user withdifferent objects or points of interest at the rest stop identified byeither other users or by a computer by analyzing the images of thescene, in one embodiment. In this embodiment, the scene is apre-recorded real-world scene that was captured by a user or by a systemand made available to other users. In some embodiments, the scene may bea virtual scene provided by an interactive application, such as a gameapplication, for example. In some embodiments, the identified objects orpoints of interest are associated with tags, or are identified usingvisual or aural cues. For example, some of the points of interestidentified in the scene of a rest stop along a scenic drive may includea duck pond, a picnic area, a gas station, one or more vista points on amountain trail, play area, an over-bridge over a stream, and rest rooms.Each of these points of interest may be identified using highlights, abeacon, an outline, an arrow, flashing lights, etc., or by sound orother aural cues, when presented at the HMD. For simplicity sake, thepoints of interest are identified in FIG. 5A using reference numerals1-9. In some embodiments, the objects or points of interest in the scene(either VR or real-world scene) are automatically identified (e.g.,visually or aurally) during rendition of the VR scene at the HMD. Inother embodiments, user interaction at the HMD may be required duringthe rendition of the VR scene to cause the objects or points of interestin the VR scene to be identified visually or aurally. For example, abutton press may be required on the HMD in order for the points ofinterest associated with the current VR scene to be highlighted. Thescene rendered on the HMD may also identify pre-defined virtual areas ator near different points of interest identified in the scene. In thescene illustrated in FIG. 5A, the user wearing the HMD is viewing the VRscene from a pre-defined virtual area near point 1 that is identifiednear the pond where ducks are present.

FIG. 5B illustrates other predefined virtual areas that are identifiedaround or near the points or objects of interest, in one embodiment. Thevirtual areas may be identified based on the terrain and the nature ortype of the object or point of interest that the virtual areas areassociated with. In FIG. 5B, each of the points of interest (1-9)identified by reference numerals have a corresponding pre-definedvirtual area (1′-9′) associated with it. The pre-defined virtual areanear the pond (represented by reference numeral 1) may be the pathway 1′surrounding the pond that is accessible to a user—meaning there are nobarriers (such as trees, shrubs, structure, etc.) that can curb a user'smovement in the virtual space. Similarly, the virtual area for the gasstation (represented by reference numeral 3) may be identified to be anarea (represented by reference numeral 3′) that is in front of the gasstation that includes the gas pumps and to the sides of the gas station.Further, the pre-defined virtual areas available near the various pointsof interest are mapped to the physical space in which the user operates,in some embodiments, so that any movement in the physical space can becorrelated with the movement of the user in virtual space.

The movement of the user in the physical space may be correlated touser's actions in the VR scene, which acts to causes a change in thescene rendered on the HMD. For example, a user may initially (i.e., timet₀) start out viewing the VR scene that includes a rest stop along thescenic route illustrated in FIG. 5A, from a pre-defined area 1′ definednear point 1. While viewing the VR scene, the user's gaze may bemonitored. When the system detects that the user's gaze direction isdirected toward point 2 from time t₃-t₈ (at least a pre-defined periodof 4 seconds, for example), the user is “teleported” to the pre-definedarea 2′ that is defined around point 2 after the expiration of thepre-defined period of time (i.e., the user is teleported between time t₇and t₈). The teleporting may be abrupt or may be done gradually, and thespeed of teleporting is dependent on the user's preference, the type ofcontent being rendered, etc. The teleporting causes the image renderedat the display screen of the HMD to be adjusted so as to change the viewof the VR scene rendered at the HMD to a new view that is rendered fromthe perspective of the user at the new location defined by thepre-defined area 2′. As part of teleporting, certain portions of the VRscene around point 2 may appear larger or closer while others in thevicinity of point 2 are presented at a distance. The direction of the VRscene is based on the user's gaze direction and the VR scene ispresented from the user's perspective. After teleporting the user to thenew location 2′, the system continues to monitor the user's actions,motions and gaze direction. When it is detected the user's gazedirection has shifted to point 7, the system determines if the user'sgaze direction is on point 7 for at least the pre-defined period of time(e.g., of 4 seconds). When it is determined that the user's gaze hasbeen on point 7 for at least the pre-defined period of time, the systemteleports the user to the pre-defined area 7′ associated with point 7and renders images of the VR scene from the perspective of the user'snew location. After teleporting the user to the new location(pre-defined area 7′), when it is determined that the user has notchanged his gaze direction for a period of time, in one implementation,the user may be left at the new location till a change is detected. Inan alternate implementation, the user may be returned to the originalview by teleporting the user to the pre-defined area near point 1 aftera period of time, and the user is presented with the VR scene from thepre-defined area near point 1.

FIG. 6 illustrates a graphical representation identifying the transitionpoints for adjusting the image rendered on the display screen of the HMDof the user, based on the gaze direction of the user, in one embodiment.At time t₀, a user may be currently viewing a VR scene that includesobject A among other objects and points of interest (point 1 in thegraph). At time t₃, the system may detect that users' gaze direction isfocused on object A (point 2 in the graph) in the VR scene. The systemcontinues to monitor the user's gaze direction and at time t₄, thesystem may determine that the user's gaze direction has remained onobject A for the pre-defined period of time, and selects, highlightsobject A for a pre-defined period of time (point 3 in the graph). Insome embodiment, as part of highlighting, the object of interest or thepoint of interest may be outlined to identify the object or point ofinterest that has caught the attention of the user.

In addition to outlining or highlighting the object or point ofinterest, a request may be generated by the system and presented to theuser to obtain confirmation that the selected object or point ofinterest is indeed the object or point of interest that has grabbed theuser's interest. The confirmation may be presented for a pre-definedperiod of time. A user, in response to the request, may provideconfirmation through an action at the HMD (e.g., button press, swipe ona touch interface provided at the HMD, etc.), or using a controller(e.g., button press or swipe gesture using the controller or via audiocommand, etc.). The user action is associated with the object or pointof interest by associating the user action to a tag of the object orpoint of interest. After expiration of the pre-defined period of time orafter obtaining the confirmation from the user, if the user's gazedirection continues to remain on object A, the system generates a signalto adjust image of object A so that object A is zoomed in. The speed ofadjusting the image of object A at the display screen may be driven bythe type of content that is being rendered (e.g., high intensity videogame scene, low intensity virtual tour, etc.), type, attributes of theobject (stationary or moving) and by the user's comfort level. Forexample, if object A is a stationary object, such as a message boardillustrated in FIGS. 4A-4C, and/or the user is able to handle a fastzoom in (based on user's reaction to various content (including movingobject) rendered on the HMD), then the object A may be zoomed in fast,as illustrated by point 4 in the graph. If, however, the user prefers aslower adjustment, the object A may be zoomed in at a moderate speed, asillustrated by point 4′ or at slower speed, as illustrated by point 4″.

In some embodiments, the view presented by the HMD is adjusted so thatthe image from the VR scene is rendered in the background and the imageof the zoomed-in object provided as an overlay on top of the VR scene.In some embodiments, the VR scene with the zoomed in object A isrendered for a period of time (e.g., 3 seconds, 4 seconds, etc.) and thedisplay screen of the HMD is reverted back to view the images from theVR scene that was being shown to the user prior to the zooming in ofobject A, as illustrated by point 5. If, on the other hand, while objectA is being highlighted for a period of time that is less than thepre-defined period of time, the user's gaze direction has shifted fromobject A, the system may retain the current view of the VR scene, asillustrated by point 3′ on the dotted line of object A.

The various embodiments that have been described in detail with regardsto the image of the VR scene or the real-world environment beingrendered on a display screen of the HMD with the image of the objectbeing zoomed in may also be extended to embodiments where the image ofthe VR scene or the real-world environment are projected by the HMD ontoa display surface that is outside of the HMD. In this case, the image ofthe object that is zoomed in is also presented on the display surface onwhich the image of the VR scene or the real-world environment is beingprojected by the HMD. In alternate embodiments, when the object iszoomed-in, the zoomed-in object may be presented on the display screenof the HMD while the VR scene is projected on to the display surface orvice versa.

In another embodiment, the object that has caught the user's attention(based on the user's gaze direction) may be a moving object, object B.In such embodiment, the adjustment to the moving object B may beperformed in a manner that takes into account the speed at which theobject B is moving. As shown in FIG. 6, when it is detected that theuser's gaze direction is on moving object B (at point 10), object B isselected and highlighted, as illustrated by point 11. The user's gazedirection continues to be monitored and the object B is zoomed in, asillustrated by point 12, when it is determined the user's gaze directioncontinues to remain on object B for at least the pre-defined period oftime. The speed of zooming in on object B may take into account thespeed at which the object B is moving so as to provide an enhanced imageof object B, without causing the user any discomfort while continuing topresent object B with sufficient clarity and focus. In some embodiments,the viewing width associated with the display screen of the HMD may bedynamically adjusted based on the speed of movement of the object and/orthe head of the user wearing the HMD, when rendering the enhanced imageof object B for the user. The dynamic adjustment of the viewing width ofthe display screen is to adjust field of view presented to the user soas to reduce motion sickness and/or assist in acclimatizing the user.The adjusted object B is rendered for a period of time, as illustratedby point 13, before resuming a current view of the VR scene in whichobject B is present.

In some embodiments, the speed at which an object is zoomed in, theamount of zooming, etc., may take into account the visioncharacteristics of the user's eyes wearing the HMD. Visioncharacteristics vary from one user to another and identify opticalfactors that are used to detect anomalies in the user's eyes affectingthe clarity of vision of the user. In some cases, the visioncharacteristics are used to determine commonly occurring refractiveerrors of the eyes that can be corrected using corrective glasses orlenses. Details of these vision characteristics associated with a usercan be taken into consideration when determining the speed or amount ofzooming in of the object.

In some embodiments, the signal to adjust the image of the object mayinclude a signal to adjust focal length of the lens of the forwardfacing camera, when capturing the images of the object from a real-worldenvironment, or a signal to adjust an aperture setting of the lens so asto cause an adjustment to a depth of the object, or a signal to adjustbrightness level of the object so that the user may be able to view theobject clearly, etc. In some embodiments, the brightness level of thescreen on which the image of the object is being rendered may also beadjusted (either by enhancing and diminishing the brightness level) toenable viewing of the object when rendered on the screen.

In some embodiments, the display screen of the HMD may be split into afirst portion and a second portion. The object that has caught theinterest of the user may be rendered in the first portion and theremaining content (real-world environment or VR scene) may be renderedin the second portion. The display screen may be split horizontally,vertically, diagonally, radially, etc, or in any other direction toallow the object or point of interest adjusted by a zoom factor to berendered while at the same time allowing the user to view the VR scenewithout interruption. For example, the screen may be split in accordanceto the direction in which an object that has caused the user's attentionis moving (e.g., a thrown or bouncing ball, etc.).

In some embodiments, the view of the images of the real-worldenvironment/VR scene and the object provided by the HMD may be split, inresponse to the signal, so that the images of the real-world environmentor the VR scene is rendered on a first screen and the image of theobject is rendered on a second screen. For example, the VR scene may beprojected onto a display surface (i.e., first screen) that is externalto the HMD using a projector mechanism that is available within the HMDwhile the enhanced image of the object is rendered on a display screenof the HMD (i.e., second screen). In some embodiments, the projectormechanism may be external to the HMD but may be controlled using inputprovided at the HMD or the controller that is communicatively coupled tothe HMD. In another example, the display surface external to the HMD maybe used to project the enhanced image of the object while the image fromthe VR scene is rendered on the display screen of the HMD.

In some embodiments, user movement in the physical space is used toadjust the image in the VR scene rendered on the HMD. The user'smovement in the physical space may be controlled by the amount of spaceavailable in pre-defined area(s) in the VR scene. In some embodiments,the VR scene presented at the HMD would identify the user's relativelocation based on the detected movement and adjusts the image in the VRscene to correlate with the detected movement. To assist the user indetermining the available space in the physical world, the system mayprovide outline of objects in the physical world as an overlay on the VRscene with the user's initial position in the VR scene highlighted orpresented as a crosshair identifier or using other identificationtechniques. As the user moves around, the user's movements are reflectedin the pre-defined virtual area near the object(s) within the VR scene.When the user moves closer to a boundary of the pre-defined virtualarea, the user is provided with visual or aural warning. In someembodiments, the visual warning for a scene of a cliff of a mountain toindicate to the user that he is moving closer to the boundary.Similarly, the user's movement in the physical world may be mapped tospace available for user's movement in the physical world. The spaceavailable in the physical world may be bound by physical objects, otherusers, etc. As the user approaches any of the physical objects or otherusers, the image in the VR scene may be adjusted to provide a warning tothe user. In some embodiment, the warning may be presented as a visualline identifying the boundary of movement for the user and the user'sproximity to the boundary. In some embodiments, the intensity of thevisual line may be increased as the user approaches the boundary. Insome embodiments, instead of or in addition to the visual cue, the usermay be presented with haptic or aural warning with the intensity of suchwarnings increasing as the user approaches closer to a physical objector the boundary defined in the VR scene. For more information onproviding warnings to users based on safety conditions, reference can bemade to application Ser. No. 14/254,881, filed on Apr. 16, 2014, andentitled, “Systems and Methods for Transitioning Between TransparentMode and Non-Transparent Mode in a Head Mounted Display,” which isincorporated herein by reference in its entirety.

The various embodiments discussed herein allow the HMD to act as avirtual binocular allowing the images to be adjusted based on the gazedirection of the user. The images are adjusted in a manner that makes itappear that the user is being teleported to a new location that iscloser to an object of interest to the user. The images are presented ina manner that makes it discernible to the user.

FIG. 7 illustrates method operations for presenting an object ofinterest from a real-world environment on a screen of a head mounteddisplay (HMD), in accordance to an embodiment of the invention. Themethod begins at operation 710, wherein an image of a real-worldenvironment proximal to a user wearing the HMD is received at the HMD.The image is received from one or more forward facing cameras disposedon the face of the HMD, for example. The image is processed by aprocessor of the HMD to determine the various objects and the scenecaptured in the image.

The HMD then determines a gaze direction of the user wearing the HMD, asillustrated in operation 720. The gaze direction is determined bytracking the user's gaze using one or more cameras, such as gazedetection cameras or other inward facing image capturing devices, thatare provided on the inside surface of the HMD and directed toward one ormore both eyes of the user. The gaze direction may be identified by aset of coordinates. The captured image is forwarded by the camera(s) tothe processor of the HMD for analysis.

The HMD processor analyzes the image received from the camera anddetermines an object or point in the real-world environment that hascaptured the user's interest, as illustrated in operation 730. Theobject or point of interest is determined by matching the set ofcoordinates identified from the gaze direction with a specific area ofthe display screen and identifying the object or point rendered in thatspecific area. This is one way of identifying the object or point ofinterest and other ways of identifying the object or the point ofinterest may also be used. For example, the image of the real-worldenvironment may be analyzed to obtain the coordinates of various points,objects found in the environment and mapping the coordinates of the gazedirection to the corresponding points, objects in the environment. Themapping may be performed using a two-dimensional or a three-dimensionalrepresentation of the image of the real-world environment. The imageanalysis also determines the virtual distance at which the object orpoint of interest is being rendered.

The processor of the HMD then generates a signal to adjust a zoom factorfor lens of the one or more forward facing cameras that was engaged tocapture the image of the real-world environment, such that the image ofthe object or point of interest is zoomed in and brought into focus, asillustrated in operation 740. The zooming in causes the image of theobject to be rendered at a second virtual distance which appears closerfor the user than the original image. Further, the zooming in is done insuch a manner that the image of the object is brought into focus whilekeeping the image sharp and clear.

FIG. 8 illustrates an embodiment of an Information Service Providerarchitecture that may be used in providing access to different games.Information Service Providers (ISP) 1070 deliver a multitude ofinformation services to users 1082 geographically dispersed andconnected via network 1050. Although the various embodiments have beendiscussed with reference to providing fast access to games, theembodiments can be extended to provide one or more types of otherservices. For example, an ISP can deliver just one type of service, suchas a game, or a variety of services such as games, stock price updates,broadcast media, news, sports, gaming, etc. Additionally, the servicesoffered by each ISP may be dynamic, that is, services can be added ortaken away at any point in time. Thus, the ISP providing a particulartype of service to a particular individual can change over time. Forexample, a user may be served by an ISP in near proximity to the userwhile the user is in her home town, and the user may be served by adifferent ISP when the user travels to a different city. The home-townISP will transfer the required information and data from the user'sgaming or access profile to the new ISP through the connection module,such that the user information “follows” the user to the new city makingthe data closer to the user and easier to access. In another embodiment,a master-server relationship may be established between a master ISP,which manages the information for the user, and a server ISP thatinterfaces directly with the user under control from the master ISP. Inanother embodiment, the data is transferred from one ISP to another ISP(i.e., during switching of data center assigned to the user) as theclient moves around the world and such transfer may be based on acompatibility of services provided by the respective ISPs to make theISP in better position to service the user be the one that deliversthese services.

ISP 1070 includes Application Service Provider (ASP) 1072, whichprovides computer-based services to customers over a network. Softwareoffered using an ASP model is also sometimes called on-demand softwareor software as a service (SaaS). A simple form of providing access to aparticular application program (such as customer relationshipmanagement) is by using a standard protocol such as HTTP. Theapplication software resides on a vendor's system, for example, and isaccessed by users through a web browser using HTML, or by a specialpurpose client software provided by the vendor, or via other remoteinterface such as a thin client.

Services delivered over a wide geographical area often use cloudcomputing. Cloud computing is a style of computing in which dynamicallyscalable and often virtualized resources are provided as a service overthe Internet. Users do not need to be an expert in the technologyinfrastructure in the “cloud” that supports them. Cloud computing can bedivided into different services, such as Infrastructure as a Service(IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS).Cloud computing services often provide common business applicationsonline that are accessed from a web browser, while the software and dataare stored on the servers. The term cloud is used as a metaphor for theInternet (e.g., using servers, storage and logic), based on how theInternet is depicted in computer network diagrams and is an abstractionfor the complex infrastructure it conceals.

Further, ISP 1070 includes a Game Processing Server (GaPS) 1074 which isused by game clients to play single and multiplayer video games. Mostvideo games played over the Internet operate via a connection to a gameserver. Typically, games use a dedicated server application thatcollects data from players and distributes it to other players. This ismore efficient and effective than a peer-to-peer arrangement, but itrequires a separate server to host the server application. In anotherembodiment, the GaPS establishes communication between the players andtheir respective game-playing devices exchange information withoutrelying on the centralized GaPS.

Dedicated GaPSs are servers which run independently of the client. Suchservers are usually run on dedicated hardware located in data centers,providing more bandwidth and dedicated processing power. Dedicatedservers are the preferred method of hosting game servers for mostPC-based multiplayer games. Massively multiplayer online games run ondedicated servers usually hosted by the software company that owns thegame title, allowing them to control and update content.

Broadcast Processing Server (BPS) 1076 distributes audio or videosignals to an audience. Broadcasting to a very narrow range of audienceis sometimes called narrowcasting. The final leg of broadcastdistribution is how the signal gets to the listener or viewer, and itmay come over the air as with a radio station or TV station to anantenna and receiver, or may come through cable TV or cable radio (or“wireless cable”) via the station or directly from a network. TheInternet may also bring either radio or TV to the recipient, especiallywith multicasting allowing the signal and bandwidth to be shared.Historically, broadcasts have been delimited by a geographic region,such as national broadcasts or regional broadcast. However, with theproliferation of fast internet, broadcasts are not defined bygeographies as the content can reach almost any country in the world.

Storage Service Provider (SSP) 1078 provides computer storage space andrelated management services. SSPs also offer periodic backup andarchiving. By offering storage as a service, users can order morestorage as required. Another major advantage is that SSPs include backupservices and users will not lose all their data if their computers' harddrives fail. Further, a plurality of SSPs can have total or partialcopies of the user data, allowing users to access data in an efficientway independently of where the user is located or the device being usedto access the data. For example, a user can access personal files in thehome computer, as well as in a mobile phone while the user is on themove.

Communications Provider 1080 provides connectivity to the users. Onekind of Communications Provider is an Internet Service Provider (ISP)which offers access to the Internet. The ISP connects its customersusing a data transmission technology appropriate for delivering InternetProtocol datagrams, such as dial-up, DSL, cable modem, fiber, wirelessor dedicated high-speed interconnects. The Communications Provider canalso provide messaging services, such as e-mail, instant messaging, andSMS texting. Another type of Communications Provider is the NetworkService provider (NSP) which sells bandwidth or network access byproviding direct backbone access to the Internet. Network serviceproviders may consist of telecommunications companies, data carriers,wireless communications providers, Internet service providers, cabletelevision operators offering high-speed Internet access, etc.

Data Exchange 1088 interconnects the several modules inside ISP 1070 andconnects these modules to users 1082 via network 1086. Data Exchange1088 can cover a small area where all the modules of ISP 1070 are inclose proximity, or can cover a large geographic area when the differentmodules are geographically dispersed. For example, Data Exchange 1088can include a fast Gigabit Ethernet (or faster) within a cabinet of adata center, or an intercontinental virtual area network (VLAN).

Users 1082 access the remote services with client device 1084, whichincludes at least a CPU, a memory, a display and I/O. The client devicecan be a PC, a mobile phone, a netbook, tablet, gaming system, a PDA,etc. In one embodiment, ISP 1070 recognizes the type of device used bythe client and adjusts the communication method employed. In othercases, client devices use a standard communications method, such asHTML, to access ISP 1070.

FIG. 9 is a block diagram of a Game System 1400, according to variousembodiments of the invention. Game System 1400 is configured to providea video stream to one or more Clients 1410 via a Network 1415. TheNetwork is similar to the Network 200 illustrated in FIG. 1. Game System1400 typically includes a Video Server System 1420 and an optional gameserver 1425. Video Server System 1420 is configured to provide the videostream to the one or more Clients 1410 with a minimal quality ofservice. For example, Video Server System 1420 may receive a gamecommand that changes the state of or a point of view within a videogame, and provide Clients 1410 with an updated video stream reflectingthis change instantly with minimal lag time. The Video Server System1420 may be configured to provide the video stream in a wide variety ofalternative video formats, including formats yet to be defined. Further,the video stream may include video frames configured for presentation toa user at a wide variety of frame rates. Typical frame rates are 30frames per second, 60 frames per second, and 1420 frames per second.Although higher or lower frame rates are included in alternativeembodiments of the invention.

Clients 1410, referred to herein individually as 1410A, 1410B, etc., mayinclude head mounted displays, terminals, personal computers, gameconsoles, tablet computers, telephones, set top boxes, kiosks, wirelessdevices, digital pads, stand-alone devices, handheld game playingdevices, and/or the like. Typically, Clients 1410 are configured toreceive encoded video streams, decode the video streams, and present theresulting video to a user, e.g., a player of a game. The processes ofreceiving encoded video streams and/or decoding the video streamstypically includes storing individual video frames in a receive bufferof the client. The video streams may be presented to the user on adisplay integral to Client 1410 or on a separate device such as amonitor or television. Clients 1410 are optionally configured to supportmore than one game player. For example, a game console may be configuredto support two, three, four or more simultaneous players. Each of theseplayers may receive a separate video stream, or a single video streammay include regions of a frame generated specifically for each player,e.g., generated based on each player's point of view. Clients 1410 areoptionally geographically dispersed. The number of clients included inGame System 1400 may vary widely from one or two to thousands, tens ofthousands, or more. As used herein, the term “game player” is used torefer to a person that plays a game and the term “game playing device”is used to refer to a device used to play a game. In some embodiments,the game playing device may refer to a plurality of computing devicesthat cooperate to deliver a game experience to the user. For example, agame console and an HMD may cooperate with the video server system 1420to deliver a game viewed through the HMD. In one embodiment, the gameconsole receives the video stream from the video server system 1420, andthe game console forwards the video stream, or updates to the videostream, to the HMD for rendering.

Clients 1410 are configured to receive video streams via Network 1415.Network 1415 may be any type of communication network including, atelephone network, the Internet, wireless networks, powerline networks,local area networks, wide area networks, private networks, and/or thelike. In typical embodiments, the video streams are communicated viastandard protocols, such as TCP/IP or UDP/IP. Alternatively, the videostreams are communicated via proprietary standards.

A typical example of Clients 1410 is a personal computer comprising aprocessor, non-volatile memory, a display, decoding logic, networkcommunication capabilities, and input devices. The decoding logic mayinclude hardware, firmware, and/or software stored on a computerreadable medium. Systems for decoding (and encoding) video streams arewell known in the art and vary depending on the particular encodingscheme used.

Clients 1410 may, but are not required to, further include systemsconfigured for modifying received video. For example, a client may beconfigured to perform further rendering, to overlay one video image onanother video image, to crop a video image, and/or the like. Forexample, Clients 1410 may be configured to receive various types ofvideo frames, such as I-frames, P-frames and B-frames, and to processthese frames into images for display to a user. In some embodiments, amember of Clients 1410 is configured to perform further rendering,shading, conversion to 3-D, or like operations on the video stream. Amember of Clients 1410 is optionally configured to receive more than oneaudio or video stream. Input devices of Clients 1410 may include, forexample, a one-hand game controller, a two-hand game controller, agesture recognition system, a gaze recognition system, a voicerecognition system, a keyboard, a joystick, a pointing device, a forcefeedback device, a motion and/or location sensing device, a mouse, atouch screen, a neural interface, a camera, input devices yet to bedeveloped, and/or the like.

The video stream (and optionally audio stream) received by Clients 1410is generated and provided by Video Server System 1420. As is describedfurther elsewhere herein, this video stream includes video frames (andthe audio stream includes audio frames). The video frames are configured(e.g., they include pixel information in an appropriate data structure)to contribute meaningfully to the images displayed to the user. As usedherein, the term “video frames” is used to refer to frames includingpredominantly information that is configured to contribute to, e.g. toeffect, the images shown to the user. Most of the teachings herein withregard to “video frames” can also be applied to “audio frames.”

Clients 1410 are typically configured to receive inputs from a user.These inputs may include game commands configured to change the state ofthe video game or otherwise affect game play. The game commands can bereceived using input devices and/or may be automatically generated bycomputing instructions executing on Clients 1410. The received gamecommands are communicated from Clients 1410 via Network 1415 to VideoServer System 1420 and/or Game Server 1425. For example, in someembodiments, the game commands are communicated to Game Server 1425 viaVideo Server System 1420. In some embodiments, separate copies of thegame commands are communicated from Clients 1410 to Game Server 1425 andVideo Server System 1420. The communication of game commands isoptionally dependent on the identity of the command. Game commands areoptionally communicated from Client 1410A through a different route orcommunication channel that that used to provide audio or video streamsto Client 1410A.

Game Server 1425 is optionally operated by a different entity than VideoServer System 1420. For example, Game Server 1425 may be operated by thepublisher of a multiplayer game. In this example, Video Server System1420 is optionally viewed as a client by Game Server 1425 and optionallyconfigured to appear from the point of view of Game Server 1425 to be aprior art client executing a prior art game engine. Communicationbetween Video Server System 1420 and Game Server 1425 optionally occursvia Network 1415. As such, Game Server 1425 can be a prior artmultiplayer game server that sends game state information to multipleclients, one of which is game server system 1420. Video Server System1420 may be configured to communicate with multiple instances of GameServer 1425 at the same time. For example, Video Server System 1420 canbe configured to provide a plurality of different video games todifferent users. Each of these different video games may be supported bya different Game Server 1425 and/or published by different entities. Insome embodiments, several geographically distributed instances of VideoServer System 1420 are configured to provide game video to a pluralityof different users. Each of these instances of Video Server System 1420may be in communication with the same instance of Game Server 1425.Communication between Video Server System 1420 and one or more GameServer 1425 optionally occurs via a dedicated communication channel. Forexample, Video Server System 1420 may be connected to Game Server 1425via a high bandwidth channel that is dedicated to communication betweenthese two systems.

Video Server System 1420 comprises at least a Video Source 1430, an I/ODevice 1445, a Processor 1450, and non-transitory Storage 1455. VideoServer System 1420 may include one computing device or be distributedamong a plurality of computing devices. These computing devices areoptionally connected via a communications system such as a local areanetwork.

Video Source 1430 is configured to provide a video stream, e.g.,streaming video or a series of video frames that form a moving picture.In some embodiments, Video Source 1430 includes a video game engine andrendering logic. The video game engine is configured to receive gamecommands from a player and to maintain a copy of the state of the videogame based on the received commands. This game state includes theposition of objects in a game environment, as well as typically a pointof view. The game state may also include properties, images, colorsand/or textures of objects.

The game state is typically maintained based on game rules, as well asgame commands such as move, turn, attack, set focus to, interact, use,and/or the like. Part of the game engine is optionally disposed withinGame Server 1425. Game Server 1425 may maintain a copy of the state ofthe game based on game commands received from multiple players usinggeographically disperse clients. In these cases, the game state isprovided by Game Server 1425 to Video Source 1430, wherein a copy of thegame state is stored and rendering is performed. Game Server 1425 mayreceive game commands directly from Clients 1410 via Network 1415,and/or may receive game commands via Video Server System 1420.

Video Source 1430 typically includes rendering logic, e.g., hardware,firmware, and/or software stored on a computer readable medium such asStorage 1455. This rendering logic is configured to create video framesof the video stream based on the game state. All or part of therendering logic is optionally disposed within a graphics processing unit(GPU). Rendering logic typically includes processing stages configuredfor determining the three-dimensional spatial relationships betweenobjects and/or for applying appropriate textures, etc., based on thegame state and viewpoint. The rendering logic produces raw video that isthen usually encoded prior to communication to Clients 1410. Forexample, the raw video may be encoded according to an Adobe Flash®standard, .wav, H.264, H.263, On2, VP6, VC-1, WMA, Huffyuv, Lagarith,MPG-x. Xvid. FFmpeg, ×264, VP6-8, realvideo, mp3, or the like. Theencoding process produces a video stream that is optionally packaged fordelivery to a decoder on a remote device. The video stream ischaracterized by a frame size and a frame rate. Typical frame sizesinclude 800×600, 1280×720 (e.g., 720p), 1024×768, although any otherframe sizes may be used. The frame rate is the number of video framesper second. A video stream may include different types of video frames.For example, the H.264 standard includes a “P” frame and an “I” frame.I-frames include information to refresh all macro blocks/pixels on adisplay device, while P-frames include information to refresh a subsetthereof. P-frames are typically smaller in data size than are I-frames.As used herein the term “frame size” is meant to refer to a number ofpixels within a frame. The term “frame data size” is used to refer to anumber of bytes required to store the frame.

In alternative embodiments Video Source 1430 includes a video recordingdevice such as a camera. This camera may be used to generate delayed orlive video that can be included in the video stream of a computer game.The resulting video stream optionally includes both rendered images andimages recorded using a still or video camera. Video Source 1430 mayalso include storage devices configured to store previously recordedvideo to be included in a video stream. Video Source 1430 may alsoinclude motion or positioning sensing devices configured to detectmotion or position of an object, e.g., person, and logic configured todetermine a game state or produce video-based on the detected motionand/or position.

Video Source 1430 is optionally configured to provide overlaysconfigured to be placed on other video. For example, these overlays mayinclude a command interface, log in instructions, messages to a gameplayer, images of other game players, video feeds of other game players(e.g., webcam video). In embodiments of Client 1410A including a touchscreen interface or a gaze detection interface, the overlay may includea virtual keyboard, joystick, touch pad, and/or the like. In one exampleof an overlay a player's voice is overlaid on an audio stream. VideoSource 1430 optionally further includes one or more audio sources.

In embodiments wherein Video Server System 1420 is configured tomaintain the game state based on input from more than one player, eachplayer may have a different point of view comprising a position anddirection of view. Video Source 1430 is optionally configured to providea separate video stream for each player based on their point of view.Further, Video Source 1430 may be configured to provide a differentframe size, frame data size, and/or encoding to each of Client 1410.Video Source 1430 is optionally configured to provide 3-D video.

I/O Device 1445 is configured for Video Server System 1420 to sendand/or receive information such as video, commands, requests forinformation, a game state, gaze information, device motion, devicelocation, user motion, client identities, player identities, gamecommands, security information, audio, and/or the like. I/O Device 1445typically includes communication hardware such as a network card ormodem. I/O Device 1445 is configured to communicate with Game Server1425, Network 1415, and/or Clients 1410.

Processor 1450 is configured to execute logic, e.g. software, includedwithin the various components of Video Server System 1420 discussedherein. For example, Processor 1450 may be programmed with softwareinstructions in order to perform the functions of Video Source 1430,Game Server 1425, and/or a Client Qualifier 1460. Video Server System1420 optionally includes more than one instance of Processor 1450.Processor 1450 may also be programmed with software instructions inorder to execute commands received by Video Server System 1420, or tocoordinate the operation of the various elements of Game System 1400discussed herein. Processor 1450 may include one or more hardwaredevice. Processor 1450 is an electronic processor.

Storage 1455 includes non-transitory analog and/or digital storagedevices. For example, Storage 1455 may include an analog storage deviceconfigured to store video frames. Storage 1455 may include a computerreadable digital storage, e.g. a hard drive, an optical drive, or solidstate storage. Storage 1415 is configured (e.g. by way of an appropriatedata structure or file system) to store video frames, artificial frames,a video stream including both video frames and artificial frames, audioframe, an audio stream, and/or the like. Storage 1455 is optionallydistributed among a plurality of devices. In some embodiments, Storage1455 is configured to store the software components of Video Source 1430discussed elsewhere herein. These components may be stored in a formatready to be provisioned when needed.

Video Server System 1420 optionally further comprises Client Qualifier1460. Client Qualifier 1460 is configured for remotely determining thecapabilities of a client, such as Clients 1410A or 1410B. Thesecapabilities can include both the capabilities of Client 1410A itself aswell as the capabilities of one or more communication channels betweenClient 1410A and Video Server System 1420. For example, Client Qualifier1460 may be configured to test a communication channel through Network1415.

Client Qualifier 1460 can determine (e.g., discover) the capabilities ofClient 1410A manually or automatically. Manual determination includescommunicating with a user of Client 1410A and asking the user to providecapabilities. For example, in some embodiments, Client Qualifier 1460 isconfigured to display images, text, and/or the like within a browser ofClient 1410A. In one embodiment, Client 1410A is an HMD that includes abrowser. In another embodiment, client 1410A is a game console having abrowser, which may be displayed on the HMD. The displayed objectsrequest that the user enter information such as operating system,processor, video decoder type, type of network connection, displayresolution, etc. of Client 1410A. The information entered by the user iscommunicated back to Client Qualifier 1460.

Automatic determination may occur, for example, by execution of an agenton Client 1410A and/or by sending test video to Client 1410A. The agentmay comprise computing instructions, such as java script, embedded in aweb page or installed as an add-on. The agent is optionally provided byClient Qualifier 1460. In various embodiments, the agent can find outprocessing power of Client 1410A, decoding and display capabilities ofClient 1410A, lag time reliability and bandwidth of communicationchannels between Client 1410A and Video Server System 1420, a displaytype of Client 1410A, firewalls present on Client 1410A, hardware ofClient 1410A, software executing on Client 1410A, registry entrieswithin Client 1410A, and/or the like.

Client Qualifier 1460 includes hardware, firmware, and/or softwarestored on a computer readable medium. Client Qualifier 1460 isoptionally disposed on a computing device separate from one or moreother elements of Video Server System 1420. For example, in someembodiments, Client Qualifier 1460 is configured to determine thecharacteristics of communication channels between Clients 1410 and morethan one instance of Video Server System 1420. In these embodiments theinformation discovered by Client Qualifier can be used to determinewhich instance of Video Server System 1420 is best suited for deliveryof streaming video to one of Clients 1410.

With the above embodiments in mind, it should be understood that theinvention may employ various computer-implemented operations involvingdata stored in computer systems. These operations include operationsrequiring physical manipulation of physical quantities. Any of theoperations described herein that form part of the invention are usefulmachine operations. The invention also relates to a device or anapparatus for performing these operations. The apparatus can bespecially constructed for the required purpose, or the apparatus can bea general-purpose computer selectively activated or configured by acomputer program stored in the computer. In particular, variousgeneral-purpose machines can be used with computer programs written inaccordance with the teachings herein, or it may be more convenient toconstruct a more specialized apparatus to perform the requiredoperations.

The above described invention may be practiced with other computersystem configurations including hand-held devices, microprocessorsystems, microprocessor-based or programmable consumer electronics,minicomputers, mainframe computers and the like. The invention may alsobe practiced in distributing computing environments where tasks areperformed by remote processing devices that are linked through acommunications network.

The invention can also be embodied as computer readable code on acomputer readable medium. Alternately, the computer readable code may bedownloaded from a server using the data exchange interconnects describedabove. The computer readable medium is any data storage device that canstore data which can be thereafter read by a computer system, includingan electromagnetic wave carrier. Examples of the computer readablemedium include hard drives, network attached storage (NAS), read-onlymemory, random-access memory, CD-ROMs, CD-Rs, CD-RWs, magnetic tapes,and other optical and non-optical data storage devices. The computerreadable medium can also be distributed over a network coupled computersystem so that the computer readable code is stored and executed in adistributed fashion.

Although the foregoing invention has been described in some detail forpurposes of clarity of understanding, it will be apparent that certainchanges and modifications may be practiced within the scope of theappended claims. Accordingly, the present embodiments are to beconsidered as illustrative and not restrictive, and the invention is notto be limited to the details given herein, but may be modified withinthe scope and equivalents of the appended claims.

What is claimed is:
 1. A method for presenting an object from areal-world environment on a screen of a head mounted display (HMD),comprising: receiving images of the real-world environment proximal to auser wearing the HMD, the images received from one or more forwardfacing cameras of the HMD and processed by a processor of the HMD forrendering on the screen; detecting a gaze direction of the user wearingthe HMD, using one or more gaze detecting cameras of the HMD that aredirected toward one or each eye of the user; analyzing the images toidentify an object captured in the images of the real-world environmentthat correlates with the gaze direction of the user, wherein an image ofthe object is rendered at a first virtual distance that causes theobject to appear out-of-focus for the user; and generating a signal toadjust a zoom factor for lens of the one or more forward facing camerasso as to cause the image of the object to be brought into focus, whereinadjusting the zoom factor results in the image of the object to bepresented on the screen of the HMD at a second virtual distance, whereinthe signal is generated upon determining the gaze direction of the userdirected toward the object lasts at least a pre-defined length of time,and wherein the zoom factor is adjusted to cause the object to berendered at the second virtual distance for a pre-defined period andupon expiration of the pre-defined period resuming rendering images fromthe real-world environment without the zoom factor.
 2. The method ofclaim 1, wherein the zoom factor is adjusted to account for visioncharacteristics of the user wearing the HMD so that the image of theobject is discernible to the user when rendered at the second virtualdistance.
 3. The method of claim 1, wherein the signal to adjust thezoom factor includes a signal to adjust an aperture setting of lens inthe one or more forward facing cameras so as to cause an adjustment to adepth at which the image of the object is captured by the one or moreforward facing cameras of the HMD.
 4. The method of claim 1, wherein thesignal to adjust the zoom factor includes a signal to adjust a focallength of lens of the one or more forward facing cameras when capturingthe images of the object in the real-world environment, the adjustmentto the focal length of the lens causes zooming in of the object.
 5. Themethod of claim 1, wherein the signal to adjust the zoom factor furtherincludes a signal to control a speed of zooming, wherein the speed ofzooming is controlled based on type of the images of the real-worldenvironment captured, attributes of the object, or based on the user. 6.The method of claim 1, wherein the signal further includes a signal toadjust brightness level of the image of the object when rendered on thescreen.
 7. The method of claim 1, further includes constructing athree-dimensional digital model of the real-world environment using theimages captured by the forward facing cameras of the HMD, thethree-dimensional digital model constructed by tracking different pointscaptured in multiple frames of the images and correlating the differentpoints to a three dimensional space.
 8. The method of claim 1, whereinidentifying the object further includes, outlining the object capturedin the images that correspond with the gaze direction; and receivingconfirmation on identification of the object from the user wearing theHMD.
 9. The method of claim 8, wherein the confirmation is obtainedthrough any one of a button press on the HMD, button press on acontroller communicatively connected to the HMD, selection of an optionprovided in a user interface rendered on the screen of the HMD, an audiocommand, gesture confirmation, or two or more combinations thereof. 10.The method of claim 8, wherein the confirmation of the object is basedon a tag associated with the object.
 11. The method of claim 1, furtherincludes splitting a view presented on the screen of the HMD into afirst portion and a second portion, the image of the object adjusted tozoom factor presented in the first portion and the image of thereal-world environment rendered in the second portion.
 12. The method ofclaim 1, wherein the second virtual distance is shorter than the firstvirtual distance.
 13. A method for presenting an object from areal-world environment rendered on a screen of a head mounted display(HMD), comprising: receiving images of a real-world environment proximalto a user wearing the HMD, the images received from one or more forwardfacing cameras of the HMD and processed by a processor within the HMDfor rendering on the screen of the HMD; identifying an object capturedin the images of the real-world environment that is rendered at a firstvirtual distance, wherein the first virtual distance makes the objectappear out-of-focus for the user; and generating a signal to adjust azoom factor for lens of the one or more forward facing cameras so as tocause an image of the object to be brought into focus for the user toview, the object being rendered at a second virtual distance, whereinthe zoom factor is adjusted to cause the object to be rendered at thesecond virtual distance for a pre-defined period of time and uponexpiration of the pre-defined period resuming rendering the images ofthe real-world environment without the zoom factor.
 14. The method ofclaim 13, wherein the object is identified based on a visual or auralnotification provided at the HMD, a visual cue provided on the screen ofthe HMD, or occurrence of a pre-defined event.